This document summarizes a presentation about proving the business value of learning and development initiatives. It discusses KnowledgeAdvisors, a firm that provides learning measurement solutions. It outlines various approaches to measuring the impact of learning, such as using surveys to estimate improved job performance or business results, templates to estimate the monetary value of results, and causal modeling to determine factors influencing impact. The presentation also addresses challenges in ensuring learning transfers to the job and sustaining a mature approach to measurement over time.

1. Emerging Challenges in Learning: Proving The Business Value
Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors

2. Agenda
 Overview of KnowledgeAdvisors
 Linking to shareholder value
 The convergence of talent management
 Showing business impact
 “Scrap” learning / manager support
 Performance optimization/Six Sigma
 Predict and forecast, then follow-up
 Actionable metrics
 Informal learning measurement
 Measurement maturity and sustainability
September 29, 2010 2© 2010, KnowledgeAdvisors

3. KnowledgeAdvisors & Metrics that Matter®
September 29, 2010 3© 2010, KnowledgeAdvisors
Experts in
metrics that
matter
KnowledgeAdvisors is a
human capital analytics firm.
We are the world’s largest
provider of learning and talent
measurement solutions.

4. KnowledgeAdvisors Clients
4September 29, 2010 © 2010, KnowledgeAdvisors

5. How Much of L&D Should you Spend?
 8 to 10% - what is spent by world class organizations
 5 to 7% - what is spent by leading edge organizations
 3 to 5 % - what should be spent by most organizations
 1 to 3 % - what is spent by some organizations
 0 to 1% - what is spent by most organizations
5
Source: Josh Bersin, KnowledgeAdvisors Analytics Symposium 2010
September 29, 2010 © 2010, KnowledgeAdvisors

6. Linking to Shareholder Value
Companies investing
in learning analytics
beat S&P 500 by 17%
Source: Human Capital Measurement and Its Impact on Stock Performance, KnowledgeAdvisors, 2009
In 2009…

7. Talent Management Convergence
7September 29, 2010 © 2010, KnowledgeAdvisors

8. Showing Business Impact
Smart
Sheets
Human
Capital
Approach
Advanced
Business
Results
Evaluation
Business
Impact
Templates
Actual
Results
Correlations
Typically
employs
surveys
Requires actual results
data and advanced
analysis techniques
Causal
Modeling
ComplexSimple
Low
Investment
High
Investment
Once you determine
Business Impact what
analysis is appropriate,
select a method:

9. Smile Sheets  Smart Sheets
9September 29, 2010 © 2010, KnowledgeAdvisors

10. Making Decisions With Impact Data
 Cut or improve
programs that aren’t
providing impact to
the business
“We need our measurement tools and
process now more than ever; if we
don’t have access to this data, how
will we be able to make intelligent
decisions on what to cut?”
- KnowledgeAdvisors Client
September 29, 2010 10© 2010, KnowledgeAdvisors

11. Human Capital Approach
Factor Item Amount
Estimate Given all factors, including this training, estimate how much
your job performance related to the course subject matter will
improve. 54.24%
Isolate Based on your response to the prior question, estimate how
much of the improvement will be a direct result of this training. 61.18%
Isolate What percent of your total work time requires the knowledge/
skills presented in this training? 62.65%
Adjust Bias, confidence, conservatism 65%
Training impact on performance improvement, adjusted for bias: 54.24% x 61.18% x
62.65% x 65% = 13.51%
 Uses improved job
performance as the overall
measure of business impact
 Easy to use, practical,
scalable, replicable
 Provides a financial ROI
relative to the improvement in
human capital performance
• Uses principles of estimation,
isolation, and adjustment
• Assumption is that human
capital’s value (i.e. salary) is
like any other asset that has
expected value that can
appreciate or depreciate over
time

12. Business Results Evaluation Approach
• Use specific business results as
a measure of business impact
• Uses an advanced survey to
estimate improvement in results
• Leverage technology to make
practical and scalable
• Provides a financial ROI relative
to the change in the business
result that prompted the training
• Uses principles of estimation,
isolation, and adjustment specific
to the business result
September 29, 2010 12© 2010, KnowledgeAdvisors

13. Business Results Evaluation Process
September 29, 2010 13© 2010, KnowledgeAdvisors

14. Case Example: Advanced Business Results
14September 29, 2010 © 2010, KnowledgeAdvisors
Background: ERP company
performing customer education
Business Issue: Show value of
training to customers in terms of
increased productivity and quality
Measurement Solution: Collected
12,770 end of program evaluations
within 6 months asking participants
to forecast impact
Analytics Results: 18.2%
estimated increase in productivity
from the training (adjusted down for
self-reported bias)

15. Business Impact Template
September 29, 2010 15© 2010, KnowledgeAdvisors
Estimate the
Impact of
Outside
Factors
Compare to
Cost of
Training
Monetize
Results

16. Use Business Impact Templates
16September 29, 2010 © 2009, KnowledgeAdvisors
Background: Large CPG based in
Canada, sales training
Business Issue: Visible sales
program, requested to link between
it and sales results
Analytics Solution: Used process
to evaluate sales change and
impact training had on sales
Analytics Results: 7% increase in
sales, 20% of the increase was due
to training, this was a $74,000
increase per sales person when the
training was $5000 per person

17. Use Actual Result Correlations
17September 29, 2010 © 2009, KnowledgeAdvisors
This is a method where actual results can be input or imported at an
aggregate, periodic frequency. It is then correlated in time periods
against training evaluation data for impact indicators.
The example below is from sales data and sale training.

18. Causal Modeling
 High level statistical
analysis using Structural
Equation Modeling
 Determines the key factors
that influence business
impact
 Causation not correlation
 The Causal Model can be
used to predict future
results and determine
investments
September 29, 2010 18© 2010, KnowledgeAdvisors
Instructor
Effectiveness
Worthwhile
Investment
Courseware
Quality
Individual
Learning
0.625
0.556
0.337
0.083
0.483

19. Scrap Learning/Manager Support
September 29, 2010 19
COSTS
RETURN
? %
DID NOT TRY IT
AT ALL
? %
TRIED IT TO SOME EXTENT BUT
GAVE UP
? %
TRIED IT AND
GOT POSITIVE
RESULTS
UNREALIZED VALUE
COSTS
RETURN
? %
DID NOT TRY IT
AT ALL
? %
TRIED IT TO SOME EXTENT BUT
GAVE UP
85%+
TRIED IT AND GOT POSITIVE RESULTS
Typical Training without Strong Manager Support
The preferred method of op7mizing training
Source: Robert Brinkerhoff
© 2010, KnowledgeAdvisors

20. Scrap Learning is a Real Problem
September 29, 2010 20
How much learning is applied to the job?
When is learning applied to the job?
What are the barriers to impact?
What can be done to reduce scrap? Source: KnowledgeAdvisors
© 2010, KnowledgeAdvisors

21. Performance Optimization
September 29, 2010 21© 2010, KnowledgeAdvisors

22. Six Sigma Approach to Performance Optimization
September 29, 2010 22© 2010, KnowledgeAdvisors

23. Predict and Forecast then Follow Up

24. Actionable Metrics
24
Benchmarks:
Internal or External
Goals: Actual to Goal
Trends: Time based analysis
Dashboards:
Concise Data
September 29, 2010 © 2010, KnowledgeAdvisors

25. Informal Learning Measurement
25September 29, 2010 © 2010, KnowledgeAdvisors
73% 56% 54% 46% 13%
% of respondents supporting each type of informal learning
Source: KnowledgeAdvisors Informal Learning Study, 2009
#ofrespondents
38
29 28
24
7
0
5
10
15
20
25
30
35
40
45
Mentoring &
Coaching (M&C)
Communities of
Practice (CoPs)
Virtual
Knowledge
Sharing (VKS)
Performance
Support Systems
(PSS)
Other
• Measure activity trends (ex. utilization of performance support tool)
• Quick Poll evaluation at point of use (ex. Amazon-like star rating)
• Comprehensive periodic evaluation to review quality and impact
• In research, 100% who measured had higher future budget allocations

26. Measurement Maturity
9/29/10 © 2010 KnowledgeAdvisors

27. Measurement Sustainability
9/29/10 © 2010 KnowledgeAdvisors
Processes
Standards,
Tools
Technology
Data
Utilization
• Established and consistent
processes, technologies and
communications for
measurement
• Broad base of skills in
analytics and
measurement and strong
capabilities in end-to-end
measurement methods
• Standardization for a consistent
and single point of analytics with
integration to and from feeder
systems
• Roles and accountabilities for
metrics are defined with
consistent expectations on
what should be measured.
• Strong desire to use the data for
decision making
• Maturityi n how data is reported and
used to improve end to end talent
processes
• Strong leadership for driving measurement
with governance to support it.
• Standards have been created for data collection
• Data integrity exists
• Culture supports and drives
need for standards and common
talent development processes
and measurement
3
5
8
1
2
3
4
6
7

28. Contact Info
www.knowledgeadvisors.com
Jeffrey Berk
Chief Operating Officer
+1 312 676 4411
jberk@knowledgeadvisors.com
7-Apr-2010 28© 2010, KnowledgeAdvisors