This document discusses strategies for measuring learning effectiveness. It outlines key principles for measurement including being relevant, credible, simple, consistent and efficient. It describes different levels of measurement including reaction, learning, behavior and results. It provides guidelines for evaluating learning at each level. These include assessing knowledge gained, skills developed, attitudes changed and on-the-job application of skills. The document also references the Kirkpatrick-Phillips model of evaluation and proposes instruments to measure factors like background knowledge, self-regulation, learning content, style and time on task.

2. MEASUREMENT STRATEGY CORE
PRINCIPLES
RELEVANT
CREDIBLE &
COMPELLING
SIMPLE &
CONSISTENT
EFFICIENT

3. MEASUREMENT KEY IMPACT
QUESTION
What will the
participants be
doing differently if
this programme
works?
What impact will
it have on those
around them?
What specific
business /department
KPI’s would you want
to see improve as a
result of the
programme?
What will success
look like for the
learners and the
business?
INDIVIDUAL GROUP UNIT CORPORATE

4. MEASUREMENT CATEGORY
PREDICTIVE MEASUREMENT
FORMATIVE MEASUREMENT
BASELINE MEASUREMENT
IN-PROCESS MEASUREMENT
RESTROSPECTIVE MEASUREMENT

13. KIRKPATRICK – PHILLIPS MODEL
Knowledge,
skills, & attitude
LEVEL 2
LEARNING

14. KIRKPATRICK – PHILLIPS MODEL
LEVEL 2
LEARNING
LEVEL 1
REACTION
LEVEL 3
BEHAVIOR
LEVEL 4
RESULTS
VALUE OF INFORMATION FREQUENCY OFUSE
DIFFICULTYOF
ASSESSMENT
Least valuable
mostvaluable
frequent easy
infrequent difficult

15. LEVEL 1 : REACTION
Evaluate trainee’s reactions to
the program:
Did they like the program?
Did they like the facilitators?
Did they like accommodation &
facilities?
Guidelinesfor evaluatingreaction
1. Determine what you want to find out
2. Design a form that will quantify
reactions
3. Encourage written comments &
suggestion
4. Get 100% immediate response
5. Get honest response
6. Develop acceptable standards
7. Measure reactions againts standards, &
take apropiate action
8. Communicate reaction as appropiate

16. LEVEL 2 : LEARNING
Measuring learning means
determining one or more of the
following:
What knowledge was learned?
What skills were developed or
improved?
What attitude were changed?
Guidelinesfor evaluatinglearning
1. Use control group if practical
2. Evaluate knowledge, skills and/or
attitude both before and after the
program
3. Use a paper-and-pencil test to measure
knowledge
4. Use performance test to measure skills
5. Get 100% immediate responses
6. Use the results of evaluation to take
appropriate action

17. LEVEL 3 : BEHAVIOR
• The frequency of application
of new skills/ knowledge/
attitudes (on the job)
• The effectiveness of the
skills/knowledge/attitudes
(as applied on the job)
Guidelinesfor evaluating BEHAVIOR
1. Use control group if practical
2. Allow time for behavioral change and
application to take place
3. Evaluate both before and after
program if practical
4. Survey and/or interview one or more
of the following: trainees, their
immediate supervisor, their
subordinates, and others who often
observe their behavior
5. Get 100% response or a sampling
6. Repeat the evaluation at appropiate
times
7. Consider soct versus benefits

18. LEVEL 4 : RESULT
• Indicate the extent to
which you think this
program has influenced
each of these measures in
your work unit,
department, or busines
unit:
• Productivity / quality /
customer response time /
cost control / employee
satisfaction / customer
satisfaction
Guidelinesfor evaluating RESULT
1. Use control group if practical
2. Allow time results to be achieved
3. Evaluate both before and after program
if practical
4. Repeat the evaluation at appropiate
times
5. Consider soct versus benefits
6. Be satisfied with the evidence if prrof is
not possible

19. MEASUREMENT OF LEARNING EFFECTIVENESS
ITEM INSTRUMENT ; REMARKS
Background knowledge, prior
skills, abilities
Conduct a pre-test
Self-regulation in learning
context (Toering, Elferink-
Gemser, Jonker, Van Heuvelen, &
Visscher, 2012)
Use of tools that support self-regulation; Collect Self-regulation of Learning and Self-Report
Scale prior and post learning intervention. The online form is available at
https://docs.google.com/spreadsheet/viewform?formkey=dHFwNnBiRjY1QWM0eE5ielA5X0pVd
Xc6MQ.
Learning content Analysis of learning content provided for the learner;
Learning support tools Use of auxiliary support tools online, on the device; usage by the learner; measurement of
learner engagement; learner analytics
Learning transfer Conduct post-tests, apply knowledge to specific tasks that map to learning objectives; these
tests include, quizzes, case studies, written assignments and peer review
Learner feedback Qualitative feedback and textual analysis
Learning style/preferences Learning modality preference inventory and Kiersey Temperament Inventory. It is anticipated
that the learning style has second order effects on learning. (see also (Neuhauser, 2002)).
Learning style may have stronger influence on learner engagement and attention span.
Time on task Record of tasks completed and the use of tools. This data may be collected on the device (aka
MoLE).
Instructional method Types used and the nature of the tasks
Teacher effects Controlled experiment

25. CAUSAL CHAIN DIAGRAM

26. Thank you
JOKO SUSENO
CORPORATE CULTURE MANAGEMENT
joko.suseno@live.com