This document discusses how to better measure and demonstrate the value of training programs. It suggests moving beyond simply measuring inputs and reactions to training, and instead focusing on outcomes like learning, application of skills on the job, impact on business metrics, and financial return on investment. The document provides examples of different levels of evaluation for training programs from reaction to ROI. It emphasizes the importance of aligning training objectives and evaluation with organizational needs and goals to demonstrate how training contributes to business performance.
4. New definition of value
Be balanced, with qualitative and quantitative
data
Contain financial and nonfinancial perspectives
Reflect strategic and tactical issues
Satisfy all key stakeholders
Be consistent in collection and analysis
Be grounded in conservative standards
Come from credible sources
6. Do we have a place in the table?
• Do training and development professionals
participate in business decisions?
• Do training and development professionals
participate in government strategic decisions?
7. Shifting Paradigms
Activity Based Results Based
No business need for the program Program linked to specific business needs
No assessment of performance issues Assessment of performance effectiveness
No specific measurable objectives Specific objectives for behavior and business impact
No effort to prepare program participants to
achieve results
Results expectations communicated to participants
No effort to prepare the work environment to
support transfer
Environment prepared to support transfer
No efforts to build partnerships with key
managers
Partnerships established with key managers and
clients
No measurement of results or benefit-cost
analysis
Measurement of results and benefit-cost analysis
Planning and reporting is input focused Planning and reporting is outcome focused
10. Levels of Evaluation
10
Levels of Evaluation Measurement Focus
0. Inputs and Indicators The input into the project in terms of
scope, volume, efficiencies, costs
Participants, Hours, Costs, Timing
1. Reaction & Perceived
Value
Reaction to the project or program,
including the perceived value
Relevance, Importance, Usefulness,
Appropriateness, Intent to use,
Motivation to take action
2. Learning
Learning to use the content and
materials, including the confidence to
use what was learned
Skills, Knowledge, Capacity,
Competencies, Confidence, Contacts
3. Application &
Implementation
Use of content and materials in the work
environment, including progress with
actual items and implementation
Extent of use, Task completion,
Frequency of use, Actions completed,
Success with use, Barriers to use,
Enablers to use
4. Business Impact
The consequences of the use of the
content and materials expressed as
business impact measures
Productivity, Revenue, Quality, Time,
Efficiency, Customer Satisfaction,
Employee Engagement
Typical Measures
5. ROI
Comparison of monetary benefits from
program to program costs
Benefit-Cost Ratio (BCR), ROI%,
Payback Period
11. Executive view – the report
Measure
We currently
measure this
We should
measure this in
the future
My ranking of the
importance of this
measure
1. Inputs: “Last year, 78,000 employees received formal learning.” 94%
2. Efficiency: “Formal learning costs $2.15 per hour of learning
consumed.”
78%
3. Reaction: “Employees rated our training very high, averaging 4.2 out
of 5.”
53%
4. Learning: “92% of participants increased knowledge and skills” 32%
5. Application: “At least 78% of employees are using the skills on the
job.”
11%
6. Impact: “Our programs are driving our top 5 business measures in
the organizations.”
8%
7. ROI: “Five ROI studies were conducted on major programs yielding an
average of 68% ROI.”
4%
8. Awards: “Our learning and development program won an award from
American Society of Training and Development.”
40%
12. Executive view – the need
Measure
We currently
measure this
We should
measure this in
the future
My ranking of the
importance of this
measure
1. Inputs: “Last year, 78,000 employees received formal learning.” 94% 85%
2. Efficiency: “Formal learning costs $2.15 per hour of learning
consumed.”
78% 82%
3. Reaction: “Employees rated our training very high, averaging 4.2 out
of 5.”
53% 22%
4. Learning: “92% of participants increased knowledge and skills” 32% 28%
5. Application: “At least 78% of employees are using the skills on the
job.”
11% 61%
6. Impact: “Our programs are driving our top 5 business measures in
the organizations.”
8% 96%
7. ROI: “Five ROI studies were conducted on major programs yielding an
average of 68% ROI.”
4% 74%
8. Awards: “Our learning and development program won an award from
American Society of Training and Development.”
40% 44%
13. Executive view – the gap
Measure
We currently
measure this
We should
measure this in
the future
My ranking of the
importance of this
measure
1. Inputs: “Last year, 78,000 employees received formal learning.” 94% 86% 6
2. Efficiency: “Formal learning costs $2.15 per hour of learning
consumed.”
78% 82% 7
3. Reaction: “Employees rated our training very high, averaging 4.2 out
of 5.”
53% 22% 8
4. Learning: “92% of participants increased knowledge and skills” 32% 28% 5
5. Application: “At least 78% of employees are using the skills on the
job.”
11% 61% 4
6. Impact: “Our programs are driving our top 5 business measures in
the organizations.”
8% 96% 1
7. ROI: “Five ROI studies were conducted on major programs yielding an
average of 68% ROI.”
4% 74% 2
8. Awards: “Our learning and development program won an award from
American Society of Training and Development.”
40% 44% 3
17. Matching evaluation levels
Improve work group productivity by 20% in 6 months 4
Initiate at least three cost reduction projects in 15 days 3
Achieve an average cost reduction of $20,000 per project 4
Use counseling skills in 90% of situations where work habits are unacceptable 3
Be able to describe the four elements of the employee assistance program 2
Provide a 4 out of 5 rating on relevance of course content 1
Achieve a 2:1 benefit-to-cost ratio one year after the new leadership program
is implemented
5
For each objective listed below, indicate the level of evaluation at which the objective is
aimed.
1. Reaction and planned action
2. Learning
3. Application
4. Impact
5. Return on Investment (ROI)
19. Key Alignment Questions
Potential ROI ROI
Payoffs Objectives5 5
Is this a problem worth solving?
Is there a potential pay off?
What is the actual ROI?
What is the BCR?
Needs Program
Assessment Objectives Evaluation
20. Needs Program
Assessment Objectives Evaluation
Business Impact Business
Needs Objectives Impact
Key Alignment Questions
4 4
What is the specific measure?
What happens if we do nothing?
Which business measure improved?
How much is related to the program?
21. Needs Program
Assessment Objectives Evaluation
Job Performance Application Application
Needs Objectives
Key Alignment Questions
3 3
What is occurring or not occurring on the
job that influences the business
measure?
What has changed?
Which skills/knowledge has been applied?
22. Skills/Knowledge Learning Learning
Needs Objectives
Needs Program
Assessment Objectives Evaluation
Key Alignment Questions
2 2
What skills or knowledge is needed to
support the job performance need?
What did they learn?
What are the takeaways?
23. Preferences Reaction Reaction &
Objectives Planned Action
Needs Program
Assessment Objectives Evaluation
Key Alignment Questions
1 1
How should the solution be
structured?
What was the reaction to the program?
How will the program be implemented?
24. Business Alignment process
1. Determine payoff needs
2. Identify business needs
3. Identify performance needs
4. Identify learning needs
5. Set Objectives
6. Design for results
7. Measure the business impact
8. Isolate the effects of the program
25. Training with the end in mind
• What is the value of your training?
– ROI
– Impact
– Application
– Learning
– Reaction
– Input
• Report the whole picture
26. Build up Capabilities in ROI
• ROI Certification program
– Sydney from 7th to 11th December 2015
– http://insources.com.au/index.php/roi/roi-
certification
27. Feedback Counts!
Your feedback helps ATD 2015 APC continue to
provide top-notch educational programs that
help you stay on top of a changing profession.
Evaluation forms for this session are available
via the mobile app and at the following link:
https://zh.surveymonkey.com/r/ATD2015APC