The document discusses the importance of measuring return on investment (ROI) in training programs to evaluate their effectiveness as training budgets increase in various regions. It outlines several key evaluation models and frameworks, such as the balanced scorecard and Kirkpatrick's four-level evaluation, emphasizing the need for systematic evaluation and accountability. Additionally, various companies are highlighted to exemplify successful ROI calculations, illustrating how training initiatives can positively impact productivity and profitability.

1. Arriffin Mansor
1

2.  Why ROI
 Identify a feasible ROI Process
 Use ROI to measure the effectiveness of
training
2

3.  Training budgets are increasing
 ROI is the ultimate level of evaluation
 Competitive pressures on costs and
productivity
 Top executives are requiring ROI information
3

4.  Commitment of Training Expenditure as a
Percentage of Payroll:
◦ US – 1% - 4% of Payroll
◦ Europe – 2.5% to 3% of Payroll
◦ Asia – 4% to 8% of Payroll
4

5. Importance : 5 – Critically important and 1 is Unimportant Level of Level of
Agreement Importance
Agreement with Trend: 5 – Very much progress and
1 – No progress
1. Training Costs are monitored more accurately to manage 3.92 4.83
resources and demonstrate accountability
2. Measuring the return on investment in training is growing in 4.02 4.71
use
3. Systematic evaluation processes measure the success of 4.57 4.69
training
4. Needs assessment and analysis is receiving more emphasis 3.76 4.64
5. Training staff and line management are forming partnerships 3.95 4.57
to achieve common goals
6. Training is linked to strategic direction of the organisation 3.96 4.48
7. The learning organisation concept is being adopted. 4.09 4.47
8. Training delivery is changing rapidly 4.26 4.39
9. Training is shifting to a performance improvement role 4.25 4.37
10. The technology of training is developing rapidly. 4.68 4.32
5

6.  ROI measures the contribution of a program/
solution designed in relation to the costs or
capital involved
 ROI = Value of Benefits – Cost of Training
Cost of Training
6

7.  Balanced Scorecard (Drs Kaplan and
Norton),
 Kirkpatrick Four-Level Framework
(Kirkpatrick, 1975),
 Jack Phillips’s Five Levels of Evaluation
7

8.  Balanced Scorecard (BSC) is a management
framework used for implementing
organisational strategy by linking the objectives,
initiatives and measures of a business.
 It integrates traditional financial measures with
other key performance indicators including
customer perspectives, internal business
processes and organisational development and
innovation.
8

9.  Level 1 Reaction Evaluation
 Level 2 Learning Evaluation
 Level 3 Transfer of Learning
Evaluation
 Level 4 Results Evaluation
This model is also adopted by Singapore Skills
Development Fund (SDF) for companies’ Training
Effectiveness Reports to SDF for funding
purposes.
9

10. Level Measurement Focus
1. Reaction & Measures participant
satisfaction with
Planned Action the program and captures planned
actions.
2. Learning Measures changes in knowledge,
skills and attitudes.
3. Application Measures changes in on-
the-job
behavior.
4. Business Impact Measures changes in business
impact
variables – any measurable
results?
5. Return on Compares program benefits
to the costs
Investment 10

11.  Singapore Airlines
 Singapore Technology
 Motorola
 Compaq
 Companies who submit to SDF for funding
11

12.  Improve program/Process
 Discontinue/Expand Programs
 Approve projects (If Pilot)
 Rank projects
 Like a front end analysis – to decide whether
worthwhile to proceed on the training project.
 Develop Data Base of Program Results
 Inform/Educate Management
 Inform/Educate target Groups
 Build Skills with Staff
12

13. Evaluation
Purposes Tabulating
Evaluation
Program
Instruments
Costs
Converting
Collecting Isolating the Data to Calculating
Post Program Effects of Monetary the Return on
Data the Program Value Investment
Significant Influences
Evaluating Policy Statement Identifying
Timing Procedures & Guidelines Intangible
Staff Skills Benefits
Management Support
Evaluating Technical Support
Levels Organisational Culture
13

14.  Interactive Selling Skills – 3 days course, 48
participants
 Retail Merchandise Company
 You may want to pay attention – you will be
tested on your understanding! – Level 2
Evaluation
14

15. Surveys,
Performance records
If management don’t
Evaluation believe in it – don’t do it
Instruments Evaluation
Purposes Tabulating
Program
Costs
*Performance Monitoring
*Questionnaire Increased Sales/Profits
*Follow-up Session
Converting
Collecting Isolating the Data to Calculating
Post Program Effects of Monetary the Return on
Data the Program Value Investment
Control Groups
Significant Influences
Evaluating Policy Statement Identifying
Timing Procedures & Guidelines Intangible
Staff Skills Benefits
Questionnaire Management Support
-3 mths later Evaluating Technical Support
Levels Organisational Culture
15

16.  Selected Data:
◦ Success with Objectives 4.3
◦ Relevance of Material 4.4
◦ Usefulness of Program 4.5
◦ Exercises/Skill Practices 3.9
◦ Overall Instructor Skill 4.1
16

17.  All Participants Demonstrated That They
Could Use The Skills Successfully
17

18.  Selected Data:
Strongly Agree Neither Disagree Strongly
Agree Agree/ Disagree
Disagree
I utilise the 78% 22% 0% 0% 0%
Skills taught
In the program
With Each Every Several At Least At Least
customer 3rd Times Once Daily Once
customer each day Weekly
Frequency of
Use of skills 52% 26% 18% 4% 0%
18

19.  Post Training Data
Weeks After Training Training Groups Control
Groups
1 $9,723 $9,698
2 $9,978 $9,720
3 $10,424 $9,812
13 $13,690 $11,572
14 $11,491 $9,683
15 $11,044 $10,092
Average for Weeks $12,075 $10,449
13, 14, 15
19

20.  46 participants were still in job after 3 months
 Ave Wkly Sales
◦ Trained Groups $12,075
◦ Untrained Groups $10,449
 Increase $1,626
 Profit Contribution 2% $32.50
 Total Weekly Improvement (x46) $1,495
 Total Annual benefits (x48wks) $71,760
20

21.  48 participants in 3 courses
 Facilitation Fees: 3 courses @ $3750 $11,250
 Program Materials: 48 @ $35/ppt $1,680
 Meals/Refreshments: 3 days @$28/ppt $4,032
 Facilities: 9 days @ $120 $1,080
 Participants Salaries Plus Benefits’ (35%) $12,442
 Coordination/Evaluation $2,500
 Total Costs $32,984
21

22. ROI =(Value of benefits –cost of training)/Cost of
training
Benefit Cost Ratio 71,760 = 2.2:1
32,984
ROI (%) = 38,776 = 118%
32,984
22

23. Evaluation
Purposes Tabulating
Evaluation
Program
Instruments
Costs
Converting
Collecting Isolating the Data to Calculating
Post Program Effects of Monetary the Return on
Data the Program Value Investment
Evaluating Significant Influences Identifying
Timing Policy Statement Intangible
Procedures & Guidelines Benefits
Evaluating Staff Skills
Levels Management Support
Technical Support
Organisational Culture
23

24.  Time spent on ROI
 Cost of conducting the measurement
 Complexity of variables in ROI
 Accuracy in measurements
 Credibility
 Lack of Skills to measure
 If staff does not see the need for ROI, it will
usually fail
 Without support from management, ROI process
will usually fail.
24

25. No need to evaluate all courses.
Egs of courses to be evaluated:
Level % Courses
4. Participant satisfaction 100%
5. Learning 70%
6. On-the-job Applications 50%
7. Results 10%
8. Return on Investment 5%
25

26. • Coca Cola – 8 half-day on supervisory skills
workshops – 1447% ROI, Benefit/Cost Ratio 15:1
• Yellow Freight System – Performance Appraisal
Course – 1115% ROI, Benefit/Coast ratio 12:1
• Litton Industries (Avionics) – Self Directed Work
Team course – Productivity increased 30%, Scrap
rate reduction 50%, 700% ROI
• Multi-Marques, Inc (Bakery) – 15 hr Supervisory
Skills Training – 215% ROI, Benefit/Cost Ratio 3.2:1
26

27.  Let’s discuss
27

28. Management Budget Cuts
Wants to see (No Results) ..And the
results Cycle Continues
Budget
Levels
Off
Minimum
Budget Training
Renewed increase Level
interest Renewed
Interest in
Training
28

29.  A Training Executive met up with an Instructional Designer to
discuss about the design of an e-learning programme for
5000 participants worldwide. This is the first time that the
company would be using e-learning. The initial investment in
the training is huge. The CEO is very excited about the
programme and has high expectations of this training. The
Training Executive suggested to forecast/measure the ROI.
 As an instructional designer, would you consider the
suggestion by the Training Executive? Why?
 As the Training Executive, what would you like to achieve
from measuring ROI?
29

30.  Cost of Technology - Initial cost of
implementing e-learning
 Newness of e-learning (is it effective &
efficient as F2F learning?)
 E-learning is not a proven process in many
organisation – need to show value now than
later
30

31.  In many e-learning projects, the client wants to
know the projected payback from the project.
 To venture into the expensive development
process without having some sense of the
payback is undesired by many clients.
 Consequently, there is tremendous pressure to
forecast ROI even if it is not very accurate.
31

32.  A major consulting firm
◦ CBT course for 7000 consultants in 50
countries. Cost of CBT training program =
$106/student. Previous F2F program cost is
$760/student. Life of program was 5 years &
savings = $4.5m
◦ What does this means to U?
 When requesting monetary investment for e-learning,
be prepared to evaluate effectiveness
 Conduct cost-benefits analysis to assess cost savings
for organisation
32

33.  Based on interviews with 300 managers in
UK
◦ Managers discouraged staff from CBT
◦ Managers recognised potential benefits of CBT
◦ 44% were unsure where CBT was less expensive
than classroom training
◦ 66% felt CBT would isolate staff
◦ What does this means to U?
 Evaluate e-learning to get management support
 Determine whether e-learning meet organisational efforts
& the identified needs
33

34.  Based on Yr 2000 ASTD State if the Industry
Report (USA)
◦ Projections for 2001 – significant majority of
organisations expect to be using multimedia
(91%), CD-ROM (87%), CBT (81%) and intranets
(77%)
◦ What does this mean to U?
 ROI will provide data to help you decide to continue to
stop existing training programmes
 ROI will help you to obtain support from the “bean
counters”
34

35.  Isolate the effects by getting before and after
training performance data.
 Performance data of with and without training
groups
35

36.  Identify key output for the job to be analyzed
 Identify the input variable for the output with
cause effect relationship
 Compute the KPI by dividing output with the
input. It is a ratio.
 Obtain before and targeted KPIs and insert
the actual KPIs in the diagram.
 Separate the value of the output based on
volume and efficiency.
36

37. Example of input output
dimension for Sales
• Output is net profits in a week RM20,000
• Input is weekly sales figure RM100,000
• The KPI is 0.20 profit per ringgit sales or return
on sales
• Insert the previous performance KPI which
may be 0.15 profit per ringgit sales
• The performance gap between the old and the
new position.
37

38.  After Profit 0.20 x 100,000 = 20,000
 Before Profit 0.15 x 100,000 = 15,000
 Performance Improvement = 5,000
 Efficiency improvement
 Volume improvement
38

39. 39

40.  Increase production units
 Increase value of sales
 Cost savings
◦ Less wastage
◦ Less time
◦ Less labour
◦ Less materials
◦ Less overheads
40

41.  Costs of Trainers
 Opportunity costs of trainees
 Designing and Planning costs of training
program
 Administration cost of training
 softwares
41

42.  Calculate the total cost of training or
intervention
 Identify the input and output variable that
determine performance.
 The relationship must be meaningful which
has high cause effect relationship
 Isolate the impact of training with the
following test, with and without, before and
after.
42