Bottom-line Evaluation: Measuring Results from Presentation Transcript
Bottom-line Evaluation: Measuring Results from Training and Performance Improvement Tuesday May 19, 2003 8:00-9:30 a.m. ASTD International Conference Donald J. Ford, Ph.D., CPT Training Education Management Redondo Beach, CA 90277
Donald J. Ford, Ph.D ., C.P.T ., is president and owner of Training Education Management , a training and performance improvement consulting firm in Redondo Beach, California. Dr. Ford also teaches graduate courses in Human Resource Development for Cal State Northridge and Antioch University, L.A. His consulting clients include: ASTD, Toyota, Nissan Motors U.S.A., Rockwell International, U.S. Marines Corps, Orange County Transportation Authority, Glendale Memorial Hospital, and National Education Corp., among others. He has worked in the field of human resource development for eighteen years, including training management positions at Southern California Gas Company, Magnavox, Allied-Signal and Texas Instruments. He possesses another eight years of experience in public education at the secondary, adult and college levels.
He holds a B.A. and M.A. in history and a Ph.D. in education, all from UCLA. He has published 35 articles and three books on topics in training, education and business. He edited a book entitled, In Action: Designing Training Programs published by ASTD in 1996. His third book entitled, Bottom-Line Training: How to Design and Implement Programs That Boost Profits, was released by Gulf Publishing in July, 1999 and forms the basis for this presentation. Dr. Ford has presented at the ASTD International Conference four times, and at ASTD L.A. conferences and workshops.
HPI Evaluation Model Business Analysis Performance Analysis Cause Analysis Intervention Selection Intervention Implementation Evaluation Evaluation Baseline Formative Evaluation Summative Evaluation Business Goals Performance Gaps Root Causes Solutions Project Management Change Management Business Measures Performance Measures Analysis Evaluation Evaluation Design Customer Satisfaction Performance and ROI HPI Outputs Evaluation Process HPI Process Evaluation Outputs
Figure 21-1: One -Shot Case Study Design T M T = training M = measurement Figure 21-2: One-Group Pre-Post Evaluation Design M = measurement (1 = Pre, 2 = Post) T = training M 1 M 2 T Figure 21-5: Pre-Post Control Group (Classic Experimental Design) M 1 M 2 Group 1: R M 1 M 2 T Group 2: R R = randomize T = training M = measurement (1 = Pre, 2 = Post)
Figure 21-7: Interrupted Time Series Design
Accident rates pre and post safety training
Sales revenues pre and post sales training
Error rates pre and post quality training
Turnover rate pre and post management training
M 2 M 5 T M 1 M 6 M 3 M 4 M 1- 6 = measures at regular intervals T = training
A Training Parable
One day in Training Land, the Senior Vice-President of the company’s largest division came to request training for all of his 2,000 employees. His reason is that the company is about to launch a new line of products and services and he fears employees will not be able to deliver these successfully without being retrained. Training needs to start in six months.
The content must include: product features and benefits, sales techniques, customer service and a customer information system.
Your assignment is to design an evaluation plan for this request.
Figure 24-1: Bottom-Line Impact of Training Strategic Growth Revenues Savings Training Cost Avoidance
Sales Training Impact Monthly Sales
What was the impact of the sales training on market share?
If the total annual market for these drugs is $10 million, what was the financial benefit of the sales training?
Figure 23-4: Productivity Gains Caused by Workplace Literacy Training Program
What conclusions can you draw from from this chart about the impact of the training?
If each percentage increase in productivity was worth an average of $5,000 what would be the total value (benefit) of this training?
Productivity Level (% of Ideal) Training Group Monthly Productivity Average Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Training Ends = Control Group Productivity Level 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Training Begins
Training Return on Investment (ROI)
Total benefits ÷ total costs
$ 750,000 total benefits
$ 187,500 in total costs
750,000/187,500 = 4:1
Company receives $4 in return for each $1 spent or a return of 400%
ROI Example $$
What is the ROI of this training?
$750,000 Indirect (lower turnover, absenteeism, higher productivity) $750,000 Direct (sales increase) Benefits $500,000 Indirect (participant salaries) $250,000 Direct (delivery) $250,000 Development Costs Category
Evaluating Behavior, Performance & Skill Transfer
Learned new skills
Desire to change
Supportive work climate
Lack of barriers
Rewards and incentives
Performance Measurement Methods
Figure 23-1: Sample Action Plan for Customer Service Training
Summary of Skills Learned
In this course, I learned the following new customer service skills:
How to greet customers.
How to identify customer needs.
How to locate customer account information in the customer database.
How to make simple changes to an account, like updating the customer’s name, address, applying payments, waiving late charges and fees.
How to handle customer complaints and document problems.
Ways the New Skills Can be Applied
Once I return to work, I can apply these new skills in the following ways:
Use the standard customer greeting I learned in class with every customer.
Ask questions of the customer, following the Inquiry Matrix I learned about in class.
Listen actively to identify the customer’s real need.
Use the Customer Database (CD) to access account number in the system.
Use the system features to update the customer’s account.
Handle any complaints by using the Customer Complaint Resolution Job Aid.
Sample Action Plan for Customer Service Training, continued
Obstacles or Barriers to Skills Application
Lack of time to practice on the job.
Waiting time for system to access account information.
Some customers do not speak English well and are hard to understand.
Some customers are already so upset by the time they call that it is difficult to solve their problem.
Enablers for Skill Application
My supervisor wants met to succeed and is willing to give me time to learn the new skills.
My co-workers are willing to help me and answer questions when they have time.
I like helping people and will try to satisfy upset customers.
The computer system is being improved to speed up access time.
Sample Action Plan for Customer Service Training, continued
Specific Steps To Be Taken and Deadlines
Use standard greeting on all calls. Tomorrow
Begin using the Inquiry Matrix on all calls. Tomorrow
Practice looking up customer information. Tomorrow
Become proficient on the Customer Database System. One month from now
Use the Customer Complaint Resolution Job Aid to One Month from now
resolve all customer complaints.
Follow up Actions to be Taken to Review Progress
Meet with my boss to discuss this action plan within one week.
Begin using the job aids immediately to help me.
Ask co-workers for help when I need it.
Register to attend another customer service class next month.
District Manager Training Pre-Post Survey
What conclusions would you draw about the training’s impact on supervisors’ job behavior?
What recommendations would you make about changes to the program based on this data?
How could you translate this data into financial results?
10% 3.4 3.1 3.7 3.7 2.9 2.5 3.3 3.0 3.8 3.2 Provides coaching and on the job training 29% 3.6 2.8 3.6 3.5 3.9 3.1 2.7 1.9 4.1 2.7 Establishes clear performance expectations and goals 21% 3.4 2.8 3.3 3.0 2.9 2.1 3.0 2.4 4.5 3.5 Communicates effectively with employees Post Pre Post Pre Post Pre Post Pre Post Pre This manager % Gain Average Colleague Employee Manager Self Question
% Payroll US Avg.=1.9% ‡
$/Employee/year US Avg.=$569 ‡
Hrs./Empl./year US Avg.= 36 ‡
Cost/Hour US Avg.=$28 ‡
% Billable Hrs. US Avg.=70% †
% Positive Course Ratings Avg.=91% †
% Learning Gain Avg.=45% †
% Behavior Change Avg.=25% †
Revenues/Employee Avg.=$100K †
Market/Book † Avg.=10:1
† Source=Author’s Study of High Tech firms
‡ Source: ASTD Training Data Book , 1996.
Training budget/ training hours
% Billable ____
Billable hours/staff hours
% Positive Ratings
Positive ratings/total ratings _________________
% Learning Gain
Post-pre / Pre-test scores
% Performance Gain
Post-pre / Pre-performance
Total market worth/ total physical assets
Training Design Job Aid
Skill Gap Analysis
Training Design Job Aid, continued
Train the Trainer
Transfer to Job
Closing comments and Q&A Donald J. Ford, Ph.D., CPT Training Education Management Redondo Beach, CA 90277 Telephone/Fax: 310-316-2240 E-mail: firstname.lastname@example.org Web: www.trainingeducationmanagement.com