evaluating of training

Welingkar’s Distance Learning Division
CHAPTER-06
Evaluating of Training
We Learn – A Continuous Learning Forum

Introduction
Training requires time , energy and money. Therefore an
organization needs to know whether the investment
made in training is being effectively and is worth the
effort. Top management is concerned with evaluation as
a process by which the effectiveness of the
organizations programs and operating procedures are
demonstrated. Supervisors are concerned more with
the “results” of training as compared by changes in
workers on the job performance.
Contd…

Evaluation is a planned process which provides specific
information about a selected session, program for the
purpose of determining value and or decision making.
Related to training evaluation is concerned with
providing information on the effectiveness of the
training activity to decision-makers who will make
decisions based on the information. There are various
models that have been developed to describe the role
of evaluation in the training process. It is important that
evaluation be a planned or systematic effort that is
built-in from the start of the training process.

Ascertaining Reactions Of the Participants
As the programme gets over, it is important to ascertain the
participants views and observations about its various
components both in terms of the contents and the training
process. The evaluation can either focus on a session-by-session
review or on the programme as a whole. The formal
format that the trainers choose to adopt for eliciting end-term
appraisal of the participants’ ranges from the relatively
simple “highlight the best and worst aspects” to a detailed
response through a properly devised questionnaire or
instrument.

Areas for Ascertaining Reactions of the Participants
An important decision that the training team takes prior
to the programme is about the specific elements it
would like to include in the evaluation exercise.
Course Objective
• How far the programme objectives are realistic.
• Clarity
• Individual understanding and response
• How far the training objectives are realized.
Contd.

Trainers Performance
• Effectiveness of the presentations.
• Skill of the trainers to use the training methods
• Trainers expertise in responding to the questions of the
participants.
• Whether the trainers attitude was supportive and
helpful.
• Their relationship with the training group.
Training Methods
• Their appropriateness tot the programme and the
modules.
• The extent to which they facilitated learning. Contd.

Training Group
• Its size and composition
• Selection procedure
• Level of the group and quality of its participation.
Time Schedule
• Duration of the programme
• Allocation of time to each module
• Was the daily schedule hectic.
• Sequencing the topics
• General flow and momentum of the programme
Contd.

Training Facilities
• Suitability of the sitting arrangement
• General ambience
• Appropriateness and quality of the equipment used
• General learning environment at the venue.
Physical Arrangements
• Appropriateness of the accommodation
• Arrangement for food etc.
Training Support Materials
• Usefulness and quality of the materials
• Their timely distribution. Contd.

Sources for Ascertaining Participants’ Reactions
• Structured Questionnaire
• Tests
• Open Forum
• Personal interviews
• Program committee meetings for ongoing evaluation
of the contents and process of the programme.
• Observations of the trainers
• Comments of the participation
Contd.

Purposes and Uses of Evaluation
The definition of evaluation implies 2 purposes:
1. Making decisions about improvements to be
made in the training program itself.
2. Making decisions about the value of the
training program in terms of whether to
continue to conduct the program.
Contd.

Specifically, evaluation can be used:
a) To determine whether the training program is accomplishing
its assigned objectives and if they were the “right”
objectives.
b) To Identify the strengths and weaknesses of training
activities.
c) To determine the cost/Benefit Ratio of the Training Program.
d) To Establish a Data base which Organization Leaders can use
to Demonstrate the Productivity and Efficiency of their
operational Procedures.
e) To Establish a Data Base which can assist Organization
Managers in Making Decision.

Evaluation of Training
After the training program has been completed and the
current training record filled in, the performance of
trainees can be evaluated and the attention of the
training staff drawn towards specific action for
improvement. Thus evaluation of training should help
to:
• Isolate areas of difficulty and suggest strategies for
overcoming them.
• Modify unrealistic training targets in the light of
actual group performance Contd.

• Determine whether motivation of trainees is of the
required level e.g. financial incentives for greater
output during training.
• Compare initial selection to actual performance
rating
• Highlight causes of absenteeism and labour turnover
during training
• Evaluate and modify instruction for future training
programme
• Formalized management training programme have a
vital role to play in the development of management
skills. Contd.

Evaluation can be four-pronged attempt.
1. Evaluation of reaction
2. Evaluation learning
3. Evaluation behavior
4. Evaluation results
Implementation of Kirkpatrick-Phillips Model
Level 1 in which evaluation is used by most organizations through
questionnaires
Level 2 evaluations can be accomplished by written or oral tests
Level 3 is Application
Level 4 is tying training to organizational impact
Level 5 A new approach, measuring returns on investment

Measuring Training Effectiveness and Impact
I. Prior to training
• The number of people that say they need it during the needs
assessment process.
• The number of people that sign up for it.
II. At the end of training
• The number of people that attend the session.
• The number of people that paid and attended.
• Customer satisfaction at the end of training
• A measurable change in knowledge
• Ability to solve a “mock” problem at the end of training
• Willingness to try to use the skill at the end of training. Contd.

III. Delayed Impact (non-job)
• Customer satisfaction at X weeks after the end of
training
• Customer satisfaction at X weeks after the training when
customers know the actual costs of training
• Retention of knowledge at X weeks after the end of
training.
• Ability to solve a “mock” problem at X weeks after the
end of training.
• Willingness to try the skill at X weeks after the training.
• The three systems followed are: 1. 360 degree
2. Performance DNA system 3. Janus performance
Management System

IV. On the job behavior change
• Trained individuals that self-report that they
changed their behavior , used the skill on the
job after the training.
• Trained individuals who’s managers report
that they changed their behavior or used the
skill after training.
• Trained individuals that actually are observed
to change their behavior or use the skill after
the training.
Contd.

V. On the job performance change
• Trained individuals who self-report that their actual
job performance changed as a result of their
changed behaviour, skill.
• Trained individuals who’s manager’s report that their
actual job performance changed as a result of their
changed behaviour/ skill.
• Trained individuals whose manager’s report that
their job performance changed either through
improved performance appraisal scores or specific
notations about the training on the performance
appraisal forms. Contd.

• Trained individuals that have observable /
measurable improvement in their actual job
performance as a result of their changed
behaviour.
• The performance of employees that are managed
by individuals that went through the training.
• Departmental performance in departments with
X % of employees that went through training ROI
of return on training dollar spent.
Contd.

(a) Other measures
• CEO / top management knowledge of/ approval of/
or satisfaction with the training program.
• Rank of training seminar in forced ranking by
managers of what factors contributed most to
productivity/ profitability improvement.
• Number of referrals to the training by those who
have previously attended the training
• Additional number of people who were trained by
those who have previously attended the training.
• Popularity of the program compared to others.
Contd.

(b) What to evaluate
One way to approach the issue of what to evaluate is to
identify what kind of information is needed. Dr.
Kirtpatrick developed four levels of evaluation. Reaction
1. Learning
2. Behaviour
3. Results
1. Reaction
Reaction is what a trainee thought of a particular
program. There are some guidelines for determining
participant reaction. Contd.

1. Design a questionnaire based on information
obtained during the need assessment phase.
2. Design the instrument so that the response can be
tabulated and quantified.
3. To obtain more honest opinions provide for the
anonymity of the participants.
4. Provide space for opinions about items that are not
covered in the questionnaire.
5. Pretest the questionnaire n a sample of participants
to determine its completeness.
Contd.

2. Learning
The trainer is concerned with measuring the
learning of principles, facts, techniques and
attitudes that were specified as training objectives.
There are many different measures of learning
performance including paper-and-pencil test,
learning curves etc. The objectives determine the
choice of the most appropriate measure. There are
several guideposts used in establishing a procedure
for measuring the amount of learning that takes
place. Contd.

1. The learning of each participant should be measured
so that quantitative results can be determined.
2. A before-and-after approach should be use so that
learning can be related to the program.
3. Learning should be measured in an objective basis.
4. A group not receiving training should be compared
with the group receiving training.
5. Where possible the evaluation results should be
analyzed statistically.
Where principles and facts are taught rather than
skills, it is more difficult to evaluate learning. Contd.

3. Behaviour
The term is used in reference to the measurement of
job performance. There are several guideposts in
evaluating training programs in terms of behaviour
changes.
1. A systematic appraisal should be made on-the-job
performance on a before-and-after basis.
2. The appraisal of performance should be made by
one or more of the following groups:
a. The person receiving the training. b. The persons
supervisor. c. The persons subordinates.
d. The persons peers. Contd.

3. A statistical analysis should be made to compare
the performance before and after and to relate
changes to the training program.
4. The post training appraisal should be made three
months or more after the training so that
trainees have an opportunity to put into practice
what they have learnt.
5. A group not receiving the training should be used
for comparison.
Contd.

4. Results
Evaluations in this level are used to relate the results
of the training program to organizational objectives.
Some of the results that could be examined are
costs, turnover, grievances and morale. Where the
objectives of the training program are closely tied to
specific organizational objectives effort should be
made to show a link between the training and the
changes called for in the organization objective.

The process of Evaluation---Measuring Change
One way of visualizing a process of evaluation is to set the
four levels of evaluation against a backdrop of a training
program. Evaluation should start before the training
program is designed. The next step is to plan the evaluation,
how is the data to be gathered. Who will perform the
evaluation? What techniques will be used? When will the
evaluation be conducted? The evaluation design needs to be
used on the objectives of the program and the criteria to be
measured. Once the plan is completed and the training
program begins, the data collection phase is initiated.

Validity and Reliability
Validity is a concept that means the degree to which an
evaluation technique or instrument measures what it
was intended to measure. Its a measure of accuracy.
Reliability is the degree to which evaluation techniques
and instruments measure a given characteristic
consistently. In order to get a long term view of a
training programs effectiveness, an instrument must
measure reliably for each session of the training
program.

Types of Evaluation Techniques and Instruments
The process of Training Evaluation
A training programme, like design, development, and
manufacture of a product passes through several
stages.
1. Pre-Training Evaluation
This is prior to the course and should cover an analyses
of the expectations of the trainee and his superior
officer. The existing level of knowledge and skills are
reassessed. It is carried out through discussions,
workshops etc. Contd.

2. Input and Delivery Evaluation
This is done concurrent to the training. It involves long
term management programme. It is done individually or
in group. Each topic, module is evaluated in terms of its
content, presentation, relevance and applicability. Using
a questionnaire, soon after the course, invariably on the
last day the participants reaction is obtained which
generally elicits information about the course inputs
and the impressions about the course.
Contd.

3. Post Training Evaluation
The objectives of training is to enhance individual
effectiveness which helps in improving organizational
performance. Thus the process of post-evaluation must
have reference to job improvement plan. This cannot be
done when the person is training , he needs to go back
to his job and demonstrate his newly acquired
knowledge and skills. After a lapse of 6-12 months the
results obtained are used for evaluating the applicability
of training.

Evaluation Of The Training Institutes
It is believed that organization climate is known to
contribute or hinder the transfers of learning to the job, the
training centre itself is expected to contribute towards
acquiring the required knowledge and skills by the trainees.
Its climate, facilities faculty resources, library are some
factors which would differentiate one centre from the other.
The Study
To study the effectiveness of the training of the in-house
training Institutes in public enterprises detailed
investigations using structured, protested questionnaires are
undertaken. Contd.

Three sets of questionnaires:
1. For the trainees
2. To the superior officers of trainees and
3. To the training centers are prepared and
administered.
These questionnaires contain information such as
relevance of the course inputs, extent of its fulfillment,
perceptions of organizations climate for effective
transfers of learning to the job, possible hindrances,
positive as well as negative factors for training in the
organization. The variables identified as relevant are
1. organizational variable. 2. Behavioral variables

Transfer of Learning
Individual Gains
To evaluate the effect of training on the individual after
6-12 months, the training persons stated the significant
areas where the individual development had taken
place. They were: 1. State of art knowledge on the
subject2. Development of the analytical skills. 3.
Refined interpersonal skills. 4. Innovation/ creativity. 5.
Management of stress.
Contd.

Transfer of learning to the organization
To evaluate whether in-house training has resulted in
enhanced organization performance, trainees were
requested to indicate the resultant benefits to public
enterprises. The non-tangible benefits are: 1. Better
management practices. 2. improved problem solving. 3. job
improvement plans. 4. better systems.
There is general evidence that systematic in-house training
has contributed to individual growth in public enterprises,
while the same level of organization improvement has not
taken place because of organization climate, PE systems and
Lack of motivation in PEs

Alternate approach in post-course evaluations
New approaches to post-course evaluation are being
explored. One such recent development is monitoring the
progress of the participant through post-course project
work. In this approach participants at the time of
nomination to the course in consultation with their
departmental heads identify a proble, of practical value to
the organization. During the program participants discuss,
reformulate and identify approaches to its solution in the
light of the knowledge and skill inputs in the training
sessions. This way they related the learning to work
situations and made innovating solutions.

Evaluation of in-house training institutes
In order to make training a continuous activity
addressing itself to specific training needs of the
organization, public enterprises have established large
number of in-house training institutes/ Centers.
There are 5 factors considered important for evaluating
the institute and methods of assigning point rating to
each.
a) Faculty Resources: The number of qualified faculty
members on full time basis at the centre, years of
experience in training are considered. Contd.

b) Infrastructure: This covers all infrastructure facilities
other than library, such as class rooms, syndicate rooms,
hostel, recreation facilities etc.
c) Library: This is the “brain” of any training institution/
Centre. Relevant books, journal, cases, audio, video etc.
d) In-house Research / Consultancy: The in-house
centers suffer from under exposure to research
development and consultancy work. Such
research/consultancy may be both internal or external.
e) Financial Support: The annual budget allocation, unit
cost per training are items under this factor.

evaluating of training

More Related Content

What's hot

Viewers also liked

Similar to evaluating of training

More from Divya Pachchigar

Recently uploaded

evaluating of training