How can one measure the return on learning investments in a meaningful way (slideshare)

EAPRIL 2016 : TRANSFORMATIONAL LAB
How can one measure the return on learning investments in a
meaningful way?

HOW CAN ONE MEASURE THE RETURN ON
LEARNING INVESTMENTS IN A MEANINGFUL
WAY?
TOM DE SCHRYVER
E: T.DESCHRYVER@UTWENTE.NL
T: +31534892076
WWW.RESEARCHGATE.NET/PROFILE/TOM_DE_SCHRYVER

INTERNET TO INTERACT
USERNAME: CCONGRESSOS
PASSWORD: EAPRIL@CCONGRESSO
4

Prepare to react
5
TXT 1
2
Internet 1
2
Twitter 1
2
Posting messages is anonymous
The text on this slide will instruct your audience on how to post. This
text will only appear once you start a free or a credit session.
Please note that the text and appearance of this slide (font, size,
color, etc.) cannot be changed.

Wat is jour first name?
6
1. Your audience's
responses will appear
here. Please feel free
to change the font,
color etc. This text
disappears after
starting your session
and slideshow.
2. Your audience's
to change the font,
disappears after
and slideshow.
3. Your audience's
to change the font,
disappears after
and slideshow.
Internet This text box will be used to describe the different message sending methods.
TXT The applicable explanations will be inserted after you have started a session.
Twitter It is possible to move, resize and modify the appearance of this text box.
# Messages:
0 (0%
correct)

 Scientific approaches
 Focus on statistical, methodological features
 Managerial approaches
 Kirkpatrick/Phillips Model (ROI)
 Succes case method
 Strategy mapping
7
EXISTING EVALUATION TOOLS FOR LEARNING
INVESTMENTS
( E.G. TRAINING, HUMAN RESOURCE AND ORGANIZATIONAL CAPABILITY DEVELOPMENT)

Prepare to vote
8
TXT 1
2
Internet 1
2
Twitter 1
2
Voting is anonymous
The text on this slide will instruct your audience on how to vote. This
text will only appear once you start a free or a credit session.
Please note that the text and appearance of this slide (font, size,
color, etc.) cannot be changed.

What is your most favorite evaluation method?
A. Kirkpatrick/ Philips ROI method
B. Success case method
C. Strategy mapping
D. Other
E. None
9
The question will open when you
start your session and slideshow.
# Votes: 0
Closed

What is your most favorite evaluation method?
10
Closed
A.
B.
C.
D.
E.
Kirkpatrick/ Philips ROI method
Success case method
Strategy mapping
Other
None
20.0%
40.0%
60.0%
80.0%
100.0%
We will set these example results to zero once
you've started your session and your slide show.
In the meantime, feel free to change the looks of
your results (e.g. the colors).

$$$
Behavior
Learning
Satisfaction
11
KIRKPATRICK PHILIPS ROI METHOD

3 steps approach:
 SATISFACTION: identify extreme success cases
 QUALITATIVE RESEARCH: Understand root causes of these success
cases in order to leverage the success factors
 QUALITATIVE RESEARCH: Understand reasons why success cases do
not always prevail and tackle these causes
12
SUCCESS CASE METHOD
BRINKERHOFF

 They tend to focus on the individual
 They tend to focus too much on the past
 They tend to focus on formal, cognitive learning
 They seem to focus on incremental change
13
SEVERAL PROBLEMS WITH EXISTING EVALUATION
TOOLS

IMPERFECTION KIRCKPATRICK PHILIPS ROI MODEL
The intent
$$$
Behavior
Learning
Satisfaction
The practice
Satisfaction
Learning
Behavior
$$$
14

 They tend to focus on the individual.
 Focus on individual, team AND organizational learning!
 They tend to evaluate on the past.
 It’s too late, focus on the future, SMART, goals!
 They tend to focus on formal, cognitive learning.
 Focus on the job behaviour, real effects of learning!
 They seem to focus on incremental change
 Dream, transform, plan, implement, evaluate, correct!
15
IDEAL PROPERTIES OF EVALUATION APPROACHES

16
STRATEGY MAP APPROACH:
DE SCHRYVER
Strategymap of Louis Van Gaal as a bondscoach
Win matches
Feel confindent/ taken seriously
Off field activities:
(cosy, family atmosphere...)
Films to support
On field activities
( tactics:
E.g. Dirk Kuyt,Tim Krul)
World Champion
Video platformWin/Fit culture

Strategy mapping is about DREAMING
17

Expectations in Belgium were quite high
18

 Discussion on the problems and ideal features of evaluation methods in
your company
 Experience how transformation can be achieved by means of strategy
mapping.
 Read SIMPAK Computer case (Brinkerhoff, 2005, page 93-95)
 Unstructured and structured questions (using shakespeak)
 Reflection on the answers (using shakespeak)
 Integrating the answers in a strategy map.
19
TWO WAYS TO PROCEED

Kirkpatrick ROI Success Case Strategy map
Main Attention Training
very simplified
context
Training priority -
Activity system
context
Activity system
priority. Training
is embedded
Target
population
Course
participant
Individual All organizational
constituents
Evaluation
moment
Feedback Feedback and
then forward
Feedforward and
then feedback
Relation to
change
- incremental transformational
20
COMPARISON OF EVALUATION SYSTEMS

 Cash cow:
 maintaining servers/ assuring effective and efficient performance
 New upgrades and new technologies to stay ahead of competition
 Service technicians are dedicated to clients (large companies
outsourcing IT)
 Learning need: service technicians need to be able to implement new
technologies and upgrade these smoothly.
 Existing training program accommodates this; yet the performance is
unclear
http://deakinprime.com/media/121745/sc-method_ahrd.pdf
http://blogs.ubc.ca/evaluation/files/2009/02/success20case20method.pdf.
21
SIMPAK COMPUTERS
TRAINING AT THE CASH COW OF A COMPUTER SERVICE PROVIDER

1. Which investments are needed for your learning department?
2. What precisely do you want to achieve with the learning department?
3. Who is critical/ Who do you really need to achieve this result?
4. What do they (the kings) need?
5. How will your operations provide it to them?
6. What would be your top priority learning investments in order to support
your critical operations?
22
QUESTIONS
SUPPOSE YOU ARE THE TRAINING MANAGER OF SIMPAK….

Which investments are needed for your
learning department?
23
1. Your audience's
to change the font,
disappears after
and slideshow.
2. Your audience's
to change the font,
disappears after
and slideshow.
3. Your audience's
to change the font,
disappears after
and slideshow.
# Messages:
0 (0%
correct)

Which investments are needed for your learning department?
 Investment in more hardware (server capacity) for training purposes
 Investment in better software (training soft. or software updates)
 Investments in more or better learning practitioners
 Investment in communication campaign to promote the learning
department
 Etc….
24
QUESTION 1

What precisely do you want to achieve
with the learning department?
25
1. Your audience's
to change the font,
disappears after
and slideshow.
2. Your audience's
to change the font,
disappears after
and slideshow.
3. Your audience's
to change the font,
disappears after
and slideshow.
# Messages:
0 (0%
correct)

What precisely do you want to achieve with the learning department?
(Mission)
 The learning department wants to make a profit.
 The learning department wants to contribute to the corporate strategy of
SIMPAK.
 The learning department wants to work as efficiently as possible?
 The learning department wants to increase the employability of all its
trainees/ service technicians.
 The learning department wants to grow.
 The learning department wants to be recognized/ legitimized by the top
management.
 Etc….
26
QUESTION 2A

What precisely do you want to achieve with the learning department?
(Vision)
 The learning department has to charge courses as cheap as possible.
 The learning department has to make courses as short as possible.
 The learning department has to satisfy the trainees during the course.
 The learning department has to reduce the waiting lists for course
participants.
 The learning department has to provide cutting edge educational
materials.
 The learning department has to update knowledge gaps of service
technicians.
 Etc…..
27
QUESTION 2B

Who is critical/ Who do you really need to
achieve this result?
28
1. Your audience's
to change the font,
disappears after
and slideshow.
2. Your audience's
to change the font,
disappears after
and slideshow.
3. Your audience's
to change the font,
disappears after
and slideshow.
# Messages:
0 (0%
correct)

Who is critical/ Who do you really need to achieve this result?
 Fundamental shift: From what (training) to whom (involvement)?
 In strategy mapping they are coined the
customers = the stakeholder(s) that is most responsible for creating value
(as measured) in the financial perspective
 I call them the KING(s).
29
NOTE TO QUESTION 3

Who is critical/ Who do you really need to achieve this result?
 CEO
 Sales manager
 Service technician
 CFO
 Direct boss
 Trainers
 Software/ hardware vendors
 Etc….
30
QUESTION 3

What do they (the KINGs) need?
31
1. Your audience's
to change the font,
disappears after
and slideshow.
2. Your audience's
to change the font,
disappears after
and slideshow.
3. Your audience's
to change the font,
disappears after
and slideshow.
# Messages:
0 (0%
correct)

What do they (the KINGs) need?
 To have a good bargain
 Not to waste time
 To be informed
 To be helped
 To have a nice time
 To have learned something relevant
 To have earned a lot of money
 Etc….
32
QUESTION 4

How will your operations provide it to
them?
33
1. Your audience's
to change the font,
disappears after
and slideshow.
2. Your audience's
to change the font,
disappears after
and slideshow.
3. Your audience's
to change the font,
disappears after
and slideshow.
# Messages:
0 (0%
correct)

How will your operations provide it to them?
 Good lunches
 Cheap prices
 High prices
 Good contact with line managers
 Relevant training content and expertise
 Efficient help
 Etc…..
34
QUESTION 5

What would be your top priority learning
investments in order to support your
critical operations (Q4)?
35
1. Your audience's
to change the font,
disappears after
and slideshow.
2. Your audience's
to change the font,
disappears after
and slideshow.
3. Your audience's
to change the font,
disappears after
and slideshow.
# Messages:
0

What would be your top priority learning investments in order to
support your critical operations?
(retake and rephrase of Q1)
 Investment in more hardware (server capacity) for training purposes
 Investment in better software (training soft. or software updates)
 Investments in more or better learning practitioners
 Investment in communication campaign to promote the learning
department
 Etc….
36
QUESTION 6

Financial Perspective
•Profitability
•Growth
•Shareholder
Value
The Vision
Customer Perspective
•Price
•Service
•Quality
Internal Perspective
Learning and Growth
•New Skills
•Continuous
Improvement
•Intellectual
Assets
“If we succeed,
how will we look to
our shareholders?”
“To achieve our
vision, what
customer needs
must we serve?”
“To satisfy our
customers and
shareholders, at which
business processes
must we excel?”
“To excel in our
processes, what
must our
organization learn?”
•Cycle Time
•Productivity
•Cost
Strategy map
Are you able to connect your answers? (Q6Q5 Q4 Q3 Q2 Q1)
37

Strategy map / the Balanced Scorecard of SouthWest
40

ROTA – Academy of a Dutch call agent
ROTA = LearningKPI(s)  internalKPI(s)  customerKPI(s)  financialKPI(s)
ROTA = LearningKPI(s) Internal KPI(s) CustomerKPI(s
)
FinancialKPI(s)
KPI? • Aantal
moodle
specialiste
n
aangewor
ven
• Aantal
licenties
voor
digitale
leeromgev
ingen
• %leermod
ules met
grading
systeem
• Aantal
leermodul
es in %
ontwikkeld
samen
met
callagents
• %
dropouts
• Gemiddeld
e
Tevredenh
eidscore
training
• Gemiddeld
e
doorlooptij
d volledig
traineeshi
p
• Gemiddeld
e
klanttevre
denheid
van call
agents na
1 jaar.
41

 Brinkerhoff, R. O. (2005). The success case method: A strategic
evaluation approach to increasing the value and effect of
training. Advances in Developing Human Resources, 7(1), 86-101.
 De Schryver, T. (2016) The strategy map as a tool to increase the return
on learning investments: learning practitioners first impressions at the
EAPRIL 2015 conference. In: European Association for Practitioner
Research on Improving Learning, EAPRIL 2015, 24-11-2015 - 27-11-
2015, Belval, luxembourg (pp. 173 - 192).
 http://www.slideshare.net/TomDeSchryver/how-can-one-measure-the-
return-on-learning-investments-in-a-meaningful-way-slideshare
42
EXTRA REFERENCES

How can one measure the return on learning investments in a meaningful way (slideshare)

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to How can one measure the return on learning investments in a meaningful way (slideshare)

Similar to How can one measure the return on learning investments in a meaningful way (slideshare) (20)

Recently uploaded

Recently uploaded (20)

How can one measure the return on learning investments in a meaningful way (slideshare)