Learning Evaluation

Wong Yew Yip
yewyip@gmail.com
Learning
Evaluation

Designed & Developed by : Wong Yew Yip ~ Updated 3 September 2021
 Learning Evaluation is to
determine whether the learning
conducted effectively meets its
original objectives
 A process of seeking and
gathering appropriate evidence of
learning
 Evaluation is key to providing the
information needed to build and
sustain learning initiatives
What is Learning Evaluation?

“ The more effectively you can measure
and prove knowledge, the more likely
you can turn learning into a tangible
corporate benefit “
“ If you want employees to walk away from a training
program with actual job skills that can translate to
benefits for the employees and the organization -
not just a certificate and a fancy binder - you need to
provide proof of effectiveness “
Quote : CLO Academy
Key Outputs of Learning Evaluation

Primary
Stakeholders
Customer
(Line
Manager)
Consumer
(Learner)
Organization
(Senior
Management)
Regulators
Business
Clients
Accreditation
/Certification
Bodies
Staff
Stakeholders in Learning Effectiveness
Learning
Centre
Secondary
Stakeholders

Take interest and an active part in the training program and
activities
Complete a personal action plan during and at the end of
the training for implementation on return to workplace, and
to put this into practice, with support from the line manager
Take interest and support the evaluation processes
LEARNING EVALUATION RESPONSIBILITIES LEARNER
Internal Stakeholders in Learning Effectiveness

 Work-needs and people identification
 Involvement in training program and evaluation development
 Support of pre-event preparation and holding briefing meetings with
the learner
 Give ongoing, and practical, support to the training program
 Hold a debriefing meeting with the learner on return to workplace to
discuss, agree or help to modify and agree action for their action plan
 Review the progress of learning implementation
 Final review of implementation success and assessment, where
possible, of the ROI
LEARNING EVALUATION RESPONSIBILITIES LINE MANAGER

 Awareness of the need and value of training to the organization
 The necessity of involving the Training Manager (or equivalent) in
senior management meetings where decisions are made about future
changes when training will be essential
 Knowledge of and support of training plans
 Active participation in events
 Requirement for evaluation to be performed and require regular
summary report
 Policy and strategic decisions based on results and ROI data
LEARNING EVALUATION RESPONSIBILITIES
SENIOR
MANAGEMENT

 Provision of any necessary pre-program work etc and program
planning
 Identification at the start of the program of the knowledge and skills
level of the trainees/learners
 Provision of training and learning resources to enable the learners to
learn within the objectives of the program and the learners' own
objectives
 Monitor the learning as the program progresses
 At the end of the program, assessment of and receipt of reports from
the learners of the learning levels achieved
 Ensure the production by the learners of an action plan to reinforce,
practise and implement learning
LEARNING CENTRE
- TRAINER

 Management of the training department and agreeing the training
needs and the program application
 Maintenance of interest and support in the planning and
implementation of the program, including a practical involvement
where required
 Introduction and maintenance of evaluation systems, and production of
regular reports for senior management
 Frequent, relevant contact with senior management
 Liaison with line managers, where necessary, in the assessment of the
training ROI
LEARNING CENTRE
- TRAINING MANAGER

Ownership of Organizational Learning
10% - LEARNING CENTRE
LEARNING
OUTCOMES
50 40 10 RULE
50% - LEARNER
• Identify Competency Gaps
• Undertake Learning
• Apply Learning
• Enable Performance
40% - SUPERVISOR
• Identify Performance Gaps
• Provide opportunity to learn
• Ensure application of Learning
10% - LEARNING CENTRE
• Provide Learning Solutions
• Maintain Learning Records

Participant
✓ Retain a copy of the Learning Agreement Form for follow through
✓ Bring along the completed Learning Agreement Form and submit to the
Course Leader on Day 1 of the course
Participant/Supervisor
✓ Discuss the benefits expected on-the job from attending the course
✓ Discuss on the support the participant will receive to apply the learning on-the-job
✓ Jointly complete the Learning Agreement Form
Participant
✓ Understand the Learning Intent and Learning Objectives
✓ List the benefits you would expect to acquire from the course
Dear Course Participant/Supervisor,
We, at ABC Learning Centre , continuously strive to ensure that course participants have a rewarding
learning experience, acquire competencies for improved performance and be able to apply them on-the-
job.
With this in mind, we would like participants and their supervisors to have a Pre-Learning Discussion :
Pre-Learning Discussion

LEARNING AGREEMENT Date :
Name of Program Dates of Program
Name of Learner Personal File No.
Designation Unit/Department
Name of Supervisor Designation
We have discussed and agreed on the Learning Intent, Learning Objectives and how the Performance Intent
will benefit the learner after the learning program
LEARNING INTENT
LEARNING
OBJECTIVES
PERFORMANCE
GOALS
…………………………..
Signature of Learner
……………………….
Signature of Supervisor
Learning Agreement

Learner
What
When
How
Learning Outcomes
Evaluation Design
Learning Design
Begin with
the end in mind
when designing Learning
Many make the mistake
of designing
Learning Evaluation last,
or even
after delivery of learning
To be effective, evaluation
cannot be an afterthought
or instructional add-on.
It needs to be embedded,
contextualized, and executed
within the learning process
When to design Learning Evaluation?

How to :
 Justify all learning expenses to organization?
 Implement learning programs that must successfully
educate employees on key objectives?
 Provide timely feedback to course designers & facilitators
to improve quality?
 Offer senior management and line managers bottom-line
justification?
 Implement twice as many programs with half the
resources last year?
 Prove that the organization is receiving real, tangible
benefits because of the existence of the Learning Centre?
Daunting Challenge of Learning Evaluation

 Management is made aware of
the extent to which training has
satisfied the operational needs
 The training function's
contribution to operating
results and the organization's
improvement is identifiable
Benefits of Learning Evaluation
 Participants are made aware of the contribution of training to
their own performances
 Employees return to their jobs with added skills, increased
morale and knowledge that their organization value them
enough to invest in their development

4 Levels of Evaluation

RESULT
BEHAVIOR
LEARNING
REACTION
Did it matter?
Did they use it?
Did they learn it?
Did they like it?
Kirkpatrick’s 4 Levels of Evaluation

Learners’ reactions and satisfaction ratings on Conditions for
Learning, with comments, for example, on the course, the
instructor, the facility, the food and other aspects of the learning
experience
Learners’ improvement in Knowledge, Skills, and Attitudes (KSA’s)
after the course. Pre and post-tests may be used to determine the
changes during training, or the assessment may address only the
KSA’s at the conclusion of the program
Learner’s ability to apply what is learned in the actual work
environment. Answers the question "Do Learners use their newly
acquired learning on the job?"
Individual Learners’ impact on their performances and the
organization itself. Answers the question "What impact has the
training achieved?" on the individual learner, team, business unit
or organization
LEVEL 1
REACTION
LEVEL 2
LEARNING
LEVEL 3
BEHAVIOR
LEVEL 4
RESULT

 Jack Phillips added this 5th Level -
Return on Investment
 Considered part of 4th Level by other
evaluation experts
 Measures Cost Benefit Analysis
 Most difficult to measure
 Numerous other factors affect ROI
Level 5 - ROI

ROI
RESULT
BEHAVIOR
LEARNING
REACTION
Did it matter?
Did they use it?
Did they learn it?
Did they like it?
Did we get our
returns?
5 Levels of
Evaluation

Learners’ reactions and satisfaction ratings on Conditions for Learning,
with comments, for example, on the course, the instructor, the facility, the food and
other aspects of the learning experience
Learners’ improvement in Knowledge, Skills, and Attitudes (KSA’s) after the course.
Pre and post-tests may be used to determine the changes during training, or the
assessment may address only the KSA’s at the conclusion of the program
Learner’s ability to apply what is learned in the actual work environment.
Answers the question "Do Learners use their newly acquired learning on the job?"
Individual Learners’ impact on their individual performances and/or the organization
itself. Answers the question "What impact has the training achieved?" on the
individual
learner, team, business unit or organization
Allows organizations to find out if a training program has been profitable, i.e to
determine if the monetary value of the results exceeds the cost of the training (ROI)
LEVEL 1
REACTION
LEVEL 2
LEARNING
LEVEL 3
BEHAVIOR
LEVEL 4
RESULT
LEVEL 5
ROI

FORMATIVE
EVALUATION
Level 1
Reaction
Level 2
Learning
SUMMATIVE
EVALUATION
Level 3
Behavior
Level 4
Result
Level 5
R O I
 Judge the worth of a program while the
program activities are forming (in
progress)
 Evaluation focuses on the process
 Also known as "internal" evaluation
 Judge the worth of a program at the
end of the program activities
(summation)
 Evaluation focuses is on the outcome
 Also known as "external" evaluation
Formative & Summative Evaluation

“ My course received a
student rating 9.8 out
of 10 ”
“ 72 % of my
students are using
what they learned on
the job ”
“ 85 % of my students
passed the end-of-
course mastery test ”
“ My course contributed to a
RM 672,000 increase in revenue
and RM 38,600 increase in profits ”
Level
1
Level
2
Level
3
Level
4
You get what you Evaluate

REACTION (Conditions for Learning)
OVERVIEW
 Measures learners’ feelings or level of satisfaction
 Does not measure actual learning that has taken
place
 Easy to measure
 End of course evaluation form on conditions of
learning completed by participants
 Supplemented by facilitator’s/trainer’s observations
LEVEL 1

DESCRIPTION
How the learners felt, and their personal reactions to the learning
experience, for example:
☺ Did the learners like and enjoy the learning experience?
☺ Did they consider the learning relevant?
☺ Was it a good use of their time?
☺ Did they like the venue, the style, timing, administration, etc?
☺ Level of participation
☺ Ease and comfort of experience
☺ Level of effort required to make the most of the learning
☺ Perceived practicability and potential for applying the learning
LEVEL 1

TOOLS & METHODS
☺ Typically 'happy sheets'
☺ Feedback forms based on subjective personal
reaction to the learning experience
☺ Verbal reaction which can be noted and analyzed
☺ Post-training surveys or questionnaires
☺ Online evaluation or grading by learners
☺ Subsequent verbal or written reports given by
learners to managers back at their jobs
LEVEL 1

RELEVANCE & PRACTICABILITY
☺ Can be done immediately the training ends
☺ Very easy to obtain reaction feedback
☺ Feedback is not expensive to gather or to analyze
for groups
☺ Important to know that learners were not upset or
disappointed
☺ Important that learners give a positive impression
when relating their experience to others who might
be deciding whether to experience the same
LEVEL 1

No. Questions
1.0 BEFORE THE LEARNING EXPERIENCE
1.1 I was given sufficient notice to attend the course.
1.2 I am aware of the learning objectives
1.3 I believe the learning will improve my job performance
1.4 I found it easy to get to the training venue
2.0 THE LEARNING EXPERIENCE
2.1 General
2.1.1 To what extent do you feel your personal learning objectives have been achieved?
2.1.2 Which of your personal objectives were not achieved, and why?
2.1.3 Which parts of the event do you feel will be most useful back at work?
2.1.4 Which parts of the event do you feel will be least useful, or not at all useful back at work?
2.1.5 Are there any subjects you would have liked to be included?
2.1.6 To what extent has your understanding of the subject improved as a result of program?
2.1.7 To what extent have your skills in the subject improved as a result of the program?
2.1.8 To what extent has the program helped to enhance your and understanding of your job?
2.1.9 How would you rate the program overall?
Conditions for Learning Sample Questions
LEVEL 1

No. Questions
2.2 The Learning Program
2.2.1 Stimulating
2.2.2 Useful for my work
2.2.3 Relevant to my work
2.2.4 Good discussions
2.2.5 Flexible structure
2.2.6 Well conducted
2.2.7 Demanding
2.2.8 Challenging
2.2.9 Well spaced out
2.2.10 Good use of time
2.2.11 Good level of activity
2.2.12 My objectives achieved
LEVEL 1

No. Questions
2.3 Program Content
2.3.1 What did you like best about the program?
2.3.2 What did you like least about the program?
2.3.3 What did you learn from the program?
2.3.4 What did you not learn from the program that you were expecting to learn?
2.3.5 What do you think should be added to the program?
2.3.6 What do you think should be dropped from the program?
2.3.7 To what extent did the program duplicate what you had learned previously
2.3.8 What are your views on the handouts given?
2.3.9 What are your views on the visual aids used?
2.3.10 Learning methods used (cases, exercises, group discussions & simulations) are effective
2.3.11 The course is detailed enough for my current job.
2.3.12 The contents are easy to understand
2.3.13 The learning materials are useful to me.
LEVEL 1

No. Questions
2.4 Trainer/Facilitator Rating
2.4.1 Knowledge of subject
2.4.2 Organization of sessions
2.4.3 Obvious preparation
2.4.4 Style and delivery
2.4.5 Responsiveness to group
2.4.6 Produce a good learning climate
2.4.7 Right balance between input sessions, activities, discussions, and videos
2.4.8 How did you feel about the length of the program?
2.4.9 To what extent was the program logically sequenced?
2.4.10 How did you feel about the pacing of the program?
2.4.11 How effective were the practical activities?
2.4.12 What was the level of time given for (a) the activities and (b) the follow-up discussion?
2.4.13 The facilitator made learning easy.
2.4.14 The facilitator was/were responsive to questions
2.4.15 There was effective usage of training aids
2.4.16 The outdoor activities are safe
2.4.17 The outdoor activities are relevant to the course objectives.
LEVEL 1

No. Questions
2.5 Learning Administration/ Location/Hotel/Accommodation/Travel
2.5.1 Was material necessary to the program provided to you prior to the program?
2.5.2 What was the level of the instructions given to you to, a) attend the program, b)
complete pre-program material, c) bring relevant material with you to the program,
d) travel to the training location?
2.5.3 Do you think the number of course participants was - Too few, Just right, Too many?
2.5.4 How do you rate the training location?
2.5.5 Training location - ease of travel
2.5.6 Seating comfort
2.5.7 Food quality
2.5.8 How do you rate the service (breaks, refreshments, meals, etc)?
2.5.9 How do you rate the Learning accommodation?
3.0 AFTER THE LEARNING EXPERIENCE
3.1 I can apply what I have learned to my current job
3.2 I would recommend this course to my colleagues/others
3.3 Has a post-training debriefing meeting been arranged with your line manager?
3.4 If 'No', will you have a post-training debriefing with your line manager?
3.5 To what extent will you be helped to implement your Action Plans: (a) by your line
manager (b) by your colleagues?
3.6 If your line manager does not perform post-program reviews and follow-up of your action plans, do
you agree that the Training Department can contact you for this purpose?
LEVEL 1

1
COURSE EVALUATION
BEFORE THE LEARNING EXPERIENCE
I believe the learning will improve my job performance
1.2
I am aware of the learning objectives
1.1
No
Yes
1.0
No
Yes
The learning objectives are achieved
2.1
The learning methods (eg lectures, discussions, activities) are effective
2.5
There are new learning concepts that I can apply in my current job
2.4
The contents are relevant to and useful for my current job
2.3
The course is well structured
2.2
THE LEARNING EXPERIENCE
The toolkits provided will help me to apply the learning in my current job
2.7
The course materials (manuals, handouts, etc) are useful
2.6
2.0
AFTER THE LEARNING EXPERIENCE
I would recommend this course to others
3.2
I can apply what I have learned in my current job
3.1
No
Yes
3.0
LEARNING ENVIRONMENT
The hotel accommodation is satisfactory
4.3
The quality of food/refreshment is satisfactory
4.2
The course venue is satisfactory
4.1
No
Yes
4.0
Please tick ( )

Dates
Course
2
COURSE EVALUATION
COMMENTS
What I like about this Course
Suggestions to improve this Course
A Typical End-of-Course Evaluation Form
LEVEL 1

0
5
10
15
20
25
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
Defect
Rate
Course Evaluation - Defect Rate
Before Learning Experience
Course Content
Course Leadership
Administration
After Learning Experience
2 0 0 3 2 0 0 4
HIGH DEFECT
RATE FROM
MAJOR CAUSES REMEDIAL ACTION SPECIFIC REMEDIAL ACTION
Before
Learning
Experience
Negative responses to “My Head has
briefed me on how this learning will
improve my job performance”
Seek help of Supervisors “Learning Agreement” implemented
Administration Negative responses to “The Food &
Refreshments are satisfactory”
Constant feedback to caterer Raise budget allocation per head
Sample Evaluation Report
LEVEL 1

OVERVIEW
 What knowledge and skills were learned –
principles, facts, techniques
 Measures learning by each learner
 Provide quantitative results of the course by using
tests and exercises
 Pre & Post tests on participants
 Evaluation results are analyzed statistically
 Comparisons can be made for different groups of
learners
LEARNING (Knowledge & Skills Test)
LEVEL 2

DESCRIPTION
The measurement of the increase in knowledge, skills
& attitudes or intellectual capability from before to
after the learning experience:
☺ Did the learners learn what was intended to be
taught?
☺ Did the learner experience what was intended for
them to experience?
☺ What is the extent of advancement or change in the
learners after the learning, in the direction or area
that was intended?
LEVEL 2

TOOLS & METHODS
☺ Typically assessments or tests before and after the training
☺ Interview or observation can be used before and after
although this is time-consuming and can be inconsistent
☺ Methods of assessment need to be closely related to the
aims of the learning
☺ Measurement and analysis is possible and easy on a group
scale
☺ Reliable, clear scoring and measurements need to be
established, so as to limit the risk of inconsistent
assessment
☺ Hard-copy, electronic, online or interview style assessments
are all possible
LEVEL 2

☺ Relatively simple to set up, but more investment
and thought required than reaction evaluation
☺ Highly relevant and clear-cut for certain learning
such as quantifiable or technical skills
☺ Less easy for more complex learning such as
attitudinal development, which is difficult to assess
☺ Cost escalates if systems are poorly designed,
which increases work required to measure and
analyze
LEVEL 2

Design of Knowledge/Skill Test Questions
LEVEL 2

Sample Pre & Post Knowledge & Skills Test
MODULE C5
No. Question Answer
1 Brainstorming allows you to:
A = Plan/Map Out or Schedule Activities
B = Collect the Right Data
C = Generate the Highest Quantity of Ideas
2 Which tool allows you to monitor the Upper and Lower Control
Limit?
A = Control Tools
B = Control Chart
C = Control Milestone
3 Which is a part of a Team Charter?
A = Goal Statement
B = Decision Making
C = Trend chart
4 Which tool allows you to make comparison on before and after?
A = Scatter Diagram
B = Paired Pareto Chart
C = Checksheet
D = Cause and Effect Diagram
5 Trend Chart allows you to:
A = Know which tools to use
B = Composition of Comparisons (ratio).
C = Show the Trend Over a Time Period
6 Which chart allows you to do Comparison of Independent
Elements?
A = Bar Chart
B = Paired Pareto
C = Line Chart
7 What tool allows you to Show the Distribution of Measurement
Data?
A = Scatter Diagram
B = Histogram
C = Paired Pareto
8 Histogram allows you to Examine Variations Collectively
A = True B = False
PRE-TEST Sheet No. Score (No. of Correct Answers)
MODULE C5
POST-TEST Sheet No. Score (No. of Correct Answers)
No. Question Answer
1 Brainstorming allows you to:
A = Plan/Map Out or Schedule Activities
B = Collect the Right Data
C = Generate the Highest Quantity of Ideas
2 Which tool allows you to monitor the Upper and Lower Control
Limit?
A = Control Tools
B = Control Chart
C = Control Milestone
3 Which is a part of a Team Charter?
A = Goal Statement
B = Decision Making
C = Trend chart
4 Which tool allows you to make comparison on before and after?
A = Scatter Diagram
B = Paired Pareto Chart
C = Checksheet
D = Cause and Effect Diagram
5 Trend Chart allows you to:
A = Know which tools to use
B = Composition of Comparisons (ratio).
C = Show the Trend Over a Time Period
6 Which chart allows you to do Comparison of Independent
Elements?
A = Bar Chart
B = Paired Pareto
C = Line Chart
7 What tool allows you to Show the Distribution of Measurement
Data?
A = Scatter Diagram
B = Histogram
C = Paired Pareto
8 Histogram allows you to Examine Variations Collectively
A = True B = False
LEVEL 2

Sample Pre & Post Knowledge & Skills Test
LEVEL 2

OVERVIEW OF REMITTANCES
APRIL 26, 2006
NO NAME PF NO PRE-TEST POST-TEST PASS % IMPROVE
1 ZAHARIAH IBRAHIM 40 70 PASS 75.0%
2 RATNA BAI 60 90 PASS 50.0%
3 MOHD AZLAN 60 100 PASS 66.7%
4 ZAINORAZAMIMY JUSOH 30 50 PASS 66.7%
5 YAP LAI CHING 30 70 PASS 133.3%
6 ELINA SHAFINAZ BT GULAM MOHD 50 70 PASS 40.0%
7 ONG HUI LIN 40 100 PASS 150.0%
8 AZITA BT AHMAD 40 60 PASS 50.0%
9 SHALIZA BT MANSOR 30 90 PASS 200.0%
10 ANIZA BT AYUB 50 80 PASS 60.0%
11 FAZLINDA AMIR 80 50 PASS -37.5%
12 SUZANA SUBOH 70 50 PASS -28.6%
LOWEST SCORE 30 50
HIGHEST SCORE 80 100
AVERAGE SCORE 48.3 73.3
% PASS (PRE TEST)
% PASS (POST TEST) 100%
% IMPROVEMENT
ABC TRAINING CENTRE
PRE & POST TEST RESULTS
Example of Pre & Post Test Results
LEVEL 2

% Score in the Tests
Credit Risk Management Module V Programme
Participants : 20 Newly Recruited Credit Officers
TEST SCORE RESULTS
87
93
63
40
77
53
0 50 100
Before
Programme
After
Programme Lowest
Average
Highest
Example of Evaluation Report
LEVEL 2

ADVANCED DOCUMENTARY CREDIT PROGRAMME
for 16 TFC & TPC Officers
% score in the Tests
0
1
2
3
4
1 - 5 6 - 10 11 -
15
16 -
20
21 -
25
26 -
30
31 -
35
36 -
40
41 -
45
46 -
50
51 -
55
56 -
60
61 -
65
66 -
70
71 -
75
76 -
80
81 -
85
86 -
90
91 -
95
96 -
100
Pre-Test Post-Test
LEVEL 2

OVERVIEW
 Measures on-the-job behavior change
 Before & After appraisal on-the-job behavior or
performance
 Usually appraisal by supervisor but can include
peers, subordinates
 Statistical analysis Before & After for comparison
and evidence of change
 Post appraisal done after a certain period of time
after training e.g. 3 months
BEHAVIOR (Behavior Change)
LEVEL 3

DESCRIPTION
The extent to which the learners applied the learning and changed
their behavior. Can be immediately and several months after the
learning, depending on the situation:
☺ Did the learners put their learning into effect when back on the
job?
☺ Were the relevant skills and knowledge used
☺ Was there noticeable and measurable change in the activity and
performance of the learners?
☺ Was the change in behavior and new level of knowledge
sustained?
☺ Is the learner aware of their change in behavior, knowledge, skill
level?
LEVEL 3

TOOLS & METHODS
☺ Observation and interview over time are required to assess
change, relevance of change, and sustainability of change
☺ Arbitrary snapshot assessments are not reliable because
people change in different ways at different times
☺ Assessments need to be designed to reduce subjective
judgment of the observer or interviewer
☺ 360-degree feedback is useful method
☺ Assessments can be designed around relevant
performance scenarios, and specific key performance
indicators
☺ Self-assessment can be useful, using carefully designed
criteria and measurements
LEVEL 3

☺ Measurement of behavior change is less easy to quantify
☺ Cooperation and skill of observers, typically line-managers,
are important factors, but difficult to control
☺ Management and analysis of ongoing subtle assessments
are difficult, and virtually impossible without a well-designed
system from the beginning
☺ Behavior change evaluation is possible given good support
and involvement from line managers or trainees, so it is
helpful to involve them from the start, and to identify
benefits for them, which links to the higher levels of
evaluation
LEVEL 3

BEHAVIOR (Behavior Change) - Example
COURSE : SELLING SKILLS Dates :
Name of Participant
BEFORE the Course
KEY KNOWLEDGE / SKILLS
ELEMENTS
AFTER the Course
1
Very
Poor
2
Poor
3
Avg.
4
Good
5
Very
Good
1
Very
Poor
2
Poor
3
Avg.
4
Good
5
Very
Good
1. 0 SELLING SKILLS & TECHNIQUES
1.1 Ability to demonstrate various selling approach or style
1.2 Ability to arouse interest & maintain interaction with
prospects
2.0 PERSUASIVE COMMUNICATION SKILLS
2.1 Ability to communicate ideas or thoughts clearly/confidently
2.2 Ability in using appropriate statements/questions to
change opinions and points or views of prospects
3. 0 HANDLING DIFFICULT CUSTOMERS & OBJECTIONS
3.1 Ability to manage and response to objections
3.2 Ability to turn sales resistance from prospects to
opportunities
4.0 POSITIVE MINDESET
4.1 Concentration and determination towards achieving set
tasks and targets
4.2 Motivation level and desire to achieve targets and goals
LEVEL 3

Customer Service Programme
for 38 Frontline Tellers at 5 Centres
General Appearance & Attire
When Serving Customers – Finer Aspects
When Serving Customers – Basic Service
Suggestive Selling
When Handling Complaints
0% 20% 40% 60% 80% 100%
or
Not Observed
Done Poorly or
Not Done At All
Done Just Nice
Done Well
A = After Programme
B = Before Programme
N/A
1
2
3
6B
6A
5B
5A
4B
4A
3B
3A
2B
2A
1B
1A
LEVEL 3

Managing Your Professional Image Programme
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
No Change Improvement Shown
ELEGANCE
COLOUR COORDINATION
POISE
BUSINESS BEHAVIOUR
BUSINESS PROTOCOL
BUSINESS LANGUAGE
POSITIVE BODY LANGUAGE
EXCHANGE BUSINESS CARDS WITH PRIDE
GOOD LISTENING SKILLS WHEN ENGAGING IN CONVERSATION
(Manner he/she carries himself/herself –
with confidence & professionalism)
(Tastefull groomed & looks
respectful & credible)
(Pleasant to the eye – the contrasts
complements outfit & personality)
(Warm, courteous & assertive)
(At ease with introductions, greetings,
handshakes, standing at attention & sitting at
work & meetings)
(Diplomatic choice of words that offers
constructive comments)
(Open hand gestures, attentive & friendly facial
expression when listening)
(Uses both hands to give out card with name facing
receiver and Uses both hands to receive card & reads
contents first before putting it aside)
(A well-placed hum, a nod & appropriate eye-
contact)
ABLE TO INCLUDE EVERYONE IN THE GROUP INTO A CONVERSATION (Even though some in the group may not be
speaking but only listening, they are included
in the conversation)
Behaviour Change Impact
LEVEL 3

OVERVIEW
 Measures desired job impact on the learner
 Objectives of training course specifically stated e.g.
increase sales, improve customer service, better
handling of difficult customers
 Impact of on-the-job improvements on individual
performance
 More difficult to measure
 Other factors may affect results
 Assessment done after an agreed period after
training
RESULTS (Job Impact)
LEVEL 4A

DESCRIPTION
The effect on the business or environment resulting from the
improved performance of the learner - it is the acid test:
☺ Measures would typically be business or organization key
performance indicators, such as volumes, values,
percentages, timescales, return on investment
☺ Include other quantifiable aspects of organizational
performance, like number of complaints, customer waiting
time, increased sales, reductions in accidents, downtime,
staff turnover, attrition, failures, wastage, non-compliance,
quality ratings, achievement of standards, etc.
LEVEL 4A

TOOLS & METHODS
☺ Many of these measures are already in place via
normal management systems and reporting
☺ The challenge is to identify which and how to relate
to learner’s input and influence
☺ Important to identify and agree accountability and
relevance with learners at the start of the learning,
so they understand what is to be measured
☺ Failure to link to learning input type and timing will
greatly reduce the ease by which results can be
attributed to learning
LEVEL 4A

☺ Individually, job impact evaluation is not particularly
difficult
☺ External factors greatly affect organizational and
business performance, which cloud the true cause
of good or poor results
LEVEL 4A

RESULTS (Job Impact) - Example
Name of Participant
No. KEY PERFORMANCE AREAS
BEFORE
the Course
(Monthly figure)
AFTER
the Course
(Monthly figure)
Remarks /
Comments
1. No. of Sales Leads generated
2. No, of sales calls
3. No. of sales interviews
4. No. of proposals for prospects
5. No. of sales closed
6. Amount of sales closed
LEVEL 4A

AVERAGE WAITING TIME (in minutes)
FOR 6 Centres in WP
0
2
4
6
8
10
12
14
16
18
20
Jan Feb Mar Apr May June July
Jln Tun HS Lee
Wisma Hangsam
Bkt Damansara
Wangsa Maju
Pandan Indah
Petaling Jaya
Customer Service
Programmes
m
i
n
u
t
e
s
6 Centres in WP with >90 %
attendance of number of
Tellers
LEVEL 4A

OVERVIEW
 Measures desired performance impact on the
learners’ business unit
 Objectives of training course specifically stated e.g.
increase sales, reduce cost, improve quality
 Impact of on-the-job improvements on business
unit performance
 More difficult to measure
 Other factors may affect results
 Assessment done after an agreed period after
training
RESULTS (Unit Performance Change)
LEVEL 4B

DESCRIPTION
The effect on the business or environment resulting from
the improved performance of the business unit,
department or group after undergoing learning together- it
is also the acid test:
☺ Measures would typically be business or organization
key performance indicators, such as volumes, values,
percentages, timescales
☺ Include other quantifiable aspects of organizational
performance, like number of complaints, staff turnover,
attrition, failures, wastage, non-compliance, quality
ratings, achievement of standards, etc.
LEVEL 4B

TOOLS & METHODS
☺ Many of these measures are already in place via normal
management systems and reporting
☺ The challenge is to identify which and how to relate to
business unit’s, department’s or group’s input and influence
☺ Important to identify and agree accountability and relevance
with business unit, department or group at the start of the
learning, so they understand what is to be measured
☺ Failure to link to learning input type and timing will greatly
reduce the ease by which results can be attributed to
learning
LEVEL 4B

☺ For an entire business unit, department or group,
it becomes more challenging, as teamwork and
support for each other become very important
☺ External factors greatly affect organizational and
business performance, which cloud the true cause
of good or poor results
LEVEL 4B

Name of Business Unit : No. of Staff :
No. KEY PERFORMANCE AREAS
No. of Months AFTER Course
1 2 3 4 5 6
1. No. of Sales Leads generated
2. No. of sales closed
3. Amount of sales closed
RESULTS (Unit Performance Change) - Example
LEVEL 4B

“Teaming to Battle” Programme
% increase in Sales Leads in 47 SSCs
0
1
2
3
4
5
6
10% 40% 70% 100% 130% 160% 190% 220% 330%
No. of SSCs
2000 %
% Increase in Sales Leads
Number
of
SSCs Example of Evaluation Report
LEVEL 4B

OVERVIEW
 Considered as part of Kirkpatrick’s Level 4
 Measures Cost Benefit Analysis
 Most difficult to measure - benefits very difficult to
quantify e.g. monetary value to service quality
 Many other factors can affect ROI
Return on Investment (ROI)
LEVEL 5

DESCRIPTION
ROI or Return On Investment, is to evaluate whether a learning
program has been profitable for the company:
☺ To determine if the monetary value of the results exceeds the cost
of the learning
☺ A difficult task to do in absolute terms due to the many aspects to
be taken into account, some of which are very difficult to quantify
☺ Benefits - the return - are notoriously tricky to pin down. What
value do you place on improved morale? Reduced stress levels?
Better competent staff? Improved time management?
☺ Thus, many training ROI assessments are necessarily best
estimates
LEVEL 5 Return on Investment (ROI)

TOOLS & METHODS
☺ Cost can be easily quantifiable – facilitator’s fees, venue,
transport, food, accommodation, opportunity cost of
learners being away from work, and other expenses
☺ Benefits, are difficult to quantify. In the absence of
absolutely quantifiable data, find something that will provide
a useful if notional indication. For example, increased
number and value of new sales, reduced absentee rates
after motivational or teambuilding program
☺ Important to identify and agree ROI measurements at the
start of the learning, so they understand what is to be
measured

Pros Cons
All related tasks are assigned a specific dollar amount.
Supervisors can then decide if the task is worth the price.
If not, it can be cut or modified.
The training program in question must be in effect for
some time before ROI can be calculated.
Once the dollar amounts are selected, it is a simple
matter of adjusting them as necessary due to increased
to decreased costs.
It takes more time and money to calculate ROI as an
evaluation than it does to conduct another form of
evaluation.
Internal and external experts can be used to complete
this plan.
It is difficult to isolate the effects of specific training or
programs and provide accurate estimates.
This strategy uses estimates from a variety of resources
(participants, supervisors, HR, senior managers, etc.) to
create a complete picture and assign monetary values.
Staff must be trained to calculate and/or interpret the
ROI of a program. It is a very intensive strategy to
implement and staff can lose steam rather quickly if
there are no specific deadlines in place for them.
There are a variety of approaches that can be used to
measure ROI and, so it is possible to incorporate it into
management, consumer, or participant evaluation plans.
There is the fear of a negative ROI. Many do not
want it proven that a training program is actually bad
for the company.
Companies are able to see if the programs they create or
the trainings they sponsor are truly having a positive
fiscal impact on their profits.
It is only a best estimate. There is no way to actually
prove ROI.
Pros & Cons of ROI Source : Encyclopedia of
Educational Technology

☺ Conducting a financial ROI assessment of learning
can be a controversial issue
☺ Jack Philips, the ROI 'guru', commented :
"Organizations should be considering
implementing ROI impact studies very selectively
on only 5 to 10 per cent of their training program,
otherwise it becomes incredibly expensive and
resource intensive."

Computation
(Benefits - Cost) / Cost X 100 = ROI percentage
The cost side of the equation is relatively straightforward. However quantifying benefits is much
more challenging
Jack J. Phillips (1996) is perhaps best known in the area of ROI evaluations. He describes a
basic formula for calculating ROI.
Step Process
1 Collect Level-4 evaluation data by asking, "Is there a change in job performance that is
positive and measurable?"
2 Isolate the effects of the training from other factors that may have contributed to the
results
3 Convert the results to monetary benefits
4 Total the cost of training
5 Compare the monetary benefits with the costs. The non-monetary benefits can be
presented as additional, though intangible, evidence of the program's success
Step 3 is probably the most difficult part of the evaluation framework as it is very subjective and
based on the opinions of stakeholders.
Calculating ROI
LEVEL 5

After attending a Sales Training
program
=
5% increase in number of sales of
product A from Learners of the
program
Determined from organization’s
sales system & product profitability
model
=
1% increase in sales of product A
is equivalent to an increased profit
of RM20,000
Cost of Sales Training program = RM45,000
ROI =
(Benefits-Cost)
-------------------- X 100
Cost
=
(100,000 – 45,000)
------------------------- X 100
45,000
= 122%
1
2
3
4
A ROI Computation Example
LEVEL 5

Credit Card Re-Engineering Disputes Workshop
Analysis of Cost Savings / Return On Investment (ROI)
a Visa In-Coming cases = 20,743
b Visa Out-going cases = 3,920
c Incoming chargeback cases which required documentation = 6,222
d Outgoing chargeback cases which required documentation = 2,352
e Total documentation (c + d) = 8,574
f
Cost Savings for 1 case : Postage Stamping (Registered)
Photocopying
Man Hour Savings
=
=
=
RM1.25
RM0.10
5 minutes
g
Total Savings to date : Postage Stamping (RM1.25 x 8,574 cases)
Photocopying (RM0.10 x 8,574 cases)
=
=
RM10,717.50
RM857.40
h Cost of Man Hours Savings : 5 minutes x 8,574 cases x RM0.07 per minute = RM3,000.90
i Total Savings from documents handling (g + h) : RM10,717.50 + RM857.40 + RM3,000.90 = RM14,575.80
J
RETURN ON INVESTMENT (ROI) : Cost Savings (14,575.80) – Investment in Workshop(12,680)
-------------------------------------------------------------------------------- x 100
Investment in Workshop (12,680)
= 14.95 %
Other Benefits
• Reduced chargeback turnaround time – from 375 days to 225 days
• Reduced waiting time on receipt of documents – by 15 days
• Reduced representment time – from 60 days to 45 days
Example of Evaluation
LEVEL 5

5-Step Learning Evaluation Process
• Who are the
Stakeholders?
• What level of
evaluation to be
conducted?
• When will the
evaluation be
conducted?
• What learning
outcomes?
• What results are
important to the
organization?
5-STEP LEARNING EVALUATION PROCESS
Establish
Evaluation
Criteria
Step 1
Establish
Evaluation
Method
Step 2
Establish
Evaluation
Tools
Step 3
Implement
Evaluation
Activities
Step 4
Analyze &
Report
Step 5
• What Data to
collect?
• Who provides
the Data?
• Develop
evaluation
procedure
• Specify
evaluation
activity
• Data Collection
Methods
• Evaluation
resources
• Questionnaires
• Tests
• Focus Groups
• Interviews
• Observations
• Performance
records
• Prepare
evaluation tools
& materials
• Carry out
evaluation
activities
• Coordinate data
collection
• Gather and
format data
generated from
assessment
activities
• Collate &
analyze data
• Interpret data
• Prepare report
• Improve learning
design or
content, if any
• Communicate
results to
stakeholders

General Guide to Learning Evaluation - 4 Levels
% of Learning
Programs to
conduct
Evaluation
RESULT
BEHAVIOR
LEARNING
REACTION
10%
25%
50%
100%

General Guide to Learning Evaluation - 5 Levels
% of Learning
Programs to
conduct
Evaluation
ROI
RESULT
BEHAVIOR
LEARNING
REACTION
10%
25%
50%
100%
5%

Level Evaluation Type
% of
Programs
No. of Programs
to be Conducted
No. to be
Evaluated
Actual No.
Evaluated
Variance
1 Conditions for Learning 100% 180 180
2 Knowledge/Skills Test 50% 180 90
3 Behavior Change 20% 180 36
4
Job Impact / Unit
Performance Change
10% 180 18
5 ROI 5% 180 9
Total
333
Learning Evaluation - Plan vs Actual

Thank You
Contact :
yewyip@gmail.com
for PowerPoint file

Learning Evaluation

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Learning Evaluation

Similar to Learning Evaluation (20)

More from Wong Yew Yip

More from Wong Yew Yip (20)

Recently uploaded

Recently uploaded (17)

Learning Evaluation