Trans-Measure: Tool for measuring training transfer effectiveness

Trans-Measure
08/27/99— Rev. 0

Trans-Measure
Transfer of Training Measurement and Improvement Tool

Developed by Bill Keeley, Westinghouse Government Environmental Services Company

© 1999, U.S. Department of Energy Carlsbad Area Office (CAO). The CAO transfers this tool and
associated non-exclusive intellectual property rights to U.S. businesses, educational institutions, non-profit
organizations, government agencies, and individual citizens at no cost to recipients. This effort is part of a
federal government initiative to ensure that U.S. taxpayers receive maximum return on investment for
government-funded research and development.

Trans-Measure

Introduction
Welcome to Trans-Measure, the transfer of training measurement and improvement tool. The purpose of
this tool is to provide training organizations with a simple, user-friendly means for systematically measuring
and improving employee transfer of training from the learning setting to the job. In addition to providing a
method for improving training, a significant side benefit of Trans-Measure is the generation of data needed
to market the value of training.

Trans-Measure is an offspring of the Transfer of Training Evaluation Model (TOTEM), developed by the
Westinghouse Electric Corporation in the mid-1990s. Westinghouse shared TOTEM with hundreds of
trainers worldwide, many of whom indicated that they use the model to evaluate the transfer of training.
Please note that TOTEM is also available at no cost through the U.S. Department of Energy Carlsbad Area
Office soft technology transfer program (visit www.t2ed.com). The problem with TOTEM is that it is
cumbersome. Trans-Measure streamlines the evaluation processes and puts them in a user-friendly format.

Trans-Measure introduces a significant element to the transfer of training equation: task importance. An
inherent weakness of TOTEM and other transfer of transfer of training evaluation models is the lack of data
and whether or not the training is worth transferring. Is value being added to the organization? Is proper
task performance critical to individual and organizational success, or is it a non-critical, nice-to-know?
Trans-Measure blends transfer of training evaluation and needs analysis methodologies to offer insight into
the valued added by training.

Instructions & Discussion
1. Determine administration methodology.

Trans-Measure can be administered on paper using interdepartmental mail, or electronically via e-mail,
the Intranet, or the Internet. Paper administration is the best method for organizations that have a
significant number of employees who do not have access to networked or on-line computers. If you
chose to administer Trans-Measure electronically, many commercial databases, surveys, and polling
applications are readily customizable to meet the needs of your organization. Note that Westinghouse
and the Department of Energy do not transfer software to support Trans-Measure administration
because they do not own the rights to these applications.

2. Set up a reminder system.

Trans-Measure calls for trainees to be surveyed six months after they complete a training course. Six
months allows trainees time to perform the tasks covered in the training course on the job. Establishing
an automated system to remind Trans-Measure administrators when to survey trainees is essential for
all but the smallest organizations; manually tracking these dates is extremely cumbersome. Fortunately,
nearly every training record administration or spreadsheet software application can perform this
function.

3. Prepare surveys.

Prepare a survey form for each course. Attachment 1 is an example of a prepared survey. Attachment
2 is a blank survey form for your use. If the course has well-written, task-level performance-based
objectives, preparing the survey can be as simple as adding the course title in the appropriate places
and cutting and pasting course objectives into Column A of the Trans-Measure survey form. If it is a

2

long course that contains many objectives, select for inclusion those objectives that your organization
considers to be key for employees to be able to perform on the job. Limit the objectives covered on the
survey form to 10-15. If the survey contains more objectives than that, employees are more likely to
answer in a response set (mark the same response for all tasks) and less likely to complete and return
the survey. If few or none of the course objectives seem to fit in Column A on the survey form, you
should step back and ask the following questions:

• Are we really teaching employees what they need to know?
• Do we have poorly-written objectives?
• Do our objectives match the course content?

Fix, accordingly. Be sure to update the course survey forms as course objectives change.

4. Distribute surveys.

Distribute the surveys six months after employees complete a course. Attachment 1 is an example of a
survey distributed by e-mail. Regardless of the survey format used, be sure to state the purpose of the
survey, indicate that participants will see the results, and keep the introduction casual and friendly.
Combined, these will elements will help increase the survey return rate. Don’ be discouraged if your
t
return rate is less than you expected. A response rate of 50 to 60% is typically satisfactory for purposes
of analysis.

5. Calculate the course transfer of training rate.

Assemble all of the completed surveys. Review the surveys, identifying (highlighting with a colored
marker works well) each task response that matches the following training transfer pattern:

Column B = No
Column C = Yes
Column D = D, W, M, or Y

For example, the following task response would be marked because it matches the training transfer
pattern:

A B C D E

These are the tasks that the Did you know Do remember how How often do you How important is it to your
organization wanted you to how to perform to perform this perform this task on success and the success of the
be able to perform following this task before task? Could you the job? organization that you perform
the class: you took the perform it today if this task effectively?
course? asked? Scale
Scale
D = Daily
W = Weekly +++ = Very Important
M = Monthly ++ = Important
Y = Yearly + = Somewhat Important
N = Never - = Not Very important
-- = Unimportant

1 Conduct preliminary Yes No Yes No D W M Y N -- - + ++ +++
interview

Divide the number of task responses that match the training transfer pattern by the total number of task
responses to determine the overall transfer of training rate for the course. For example, let’ say 12
s
employees returned surveys for the course listed in Attachment 1. Note that the course has 9 tasks

3

listed under Column A. Thus, the total number of task responses is 108 (12 completed surveys
multiplied by 9 tasks). The evaluator checks each of the task responses and finds that 37 match the
training transfer pattern. The evaluator divides the total number of task responses that match the
training transfer pattern (37) by the total number of task responses (108) to calculate the course transfer
of training rate (34%).

How does this example transfer of training rate for this course compare to others? Average rates
ranging from 10 to 40% are frequently mentioned at evaluation and measurement conferences and in
popular training literature. Westinghouse research conducted in the mid-1990s indicated an average
transfer of training rate of 38% for five field-tested courses, with a high of 53% for a business English
course and a low of 22% for a leadership course. The important thing here is not to compare your
courses to industry benchmarks, but to each other. For example, if you have seven courses in the 30%
to 40% transfer range and one course with a transfer rate of 18%, it is clear that the course with the low
transfer rate needs improvement, regardless of what the training industry average rates are. Conversely,
if you have seven courses in the 30% to 40% transfer range and one course with a transfer rate of 60%,
it would be a smart move to take an extensive look at the top performer to determine the factors that
make the course successful.

The remaining steps will provide you with important information for improving a course.

6. Calculate the course prior knowledge rate.

Assemble all of the completed surveys. Review the surveys, counting each task response that has YES
marked in Column B. Divide the number of Column B = YES responses by the total number of task
responses to determine the prior knowledge rate for the course.

The prior knowledge rate provides you with an indication of what percentage of the course’ content is
s
unnecessary: if they already know it, why train them? The prior knowledge rate for the previously
mentioned business English course was 30%. Using this information and data gathered on individual
objectives (see subsequent steps), the instructor for this course was able to eliminate some content and
expand the instruction in subject areas where employees experienced difficulty.

7. Calculate the course “didn’ stick” rate.
t

Assemble all of the completed surveys. Review the surveys, counting each task response that has NO
marked in Column C. Divide the number of Column C = No responses by the total number of task
responses to determine the “didn’ stick” rate for the course.
t

The “didn’ stick” rate provides you with an indication of what percentage of the course’ content was
t s
incomprehensible and/or unmemorable. The “didn’ stick” rate for the previously mentioned business
t
English course was 16%. Using this information and data gathered on individual objectives (see
subsequent steps), the instructor for this course was able to expand the instruction in subject areas
where employees experienced difficulty.

8. Calculate the course value-added score.

Review the surveys, assigning the following numbers for each task in which transfer of training
occurred (see step 5):

Column D: D = 5 points, W = 4 points, M = 3 points, Y = 2 (Note: A N response indicates transfer of
training did not occur. Therefore, the task would not be included in the calculation.)
Column E: +++ = 5 points, ++ = 4 points, + = 3 points, - = 1 point, -- = 0 points

4

For example, the following example would be numbered because transfer of training occurred (B= N0,
C= YES, D = D, W, M, or Y). The Column D response would be numbered a “2” and the Column E
response would be numbered a “3.”

A B C D E

Scale
D = Daily
-- = Unimportant

interview

Add up all of the numbers in the survey to obtain a raw score. Divide the raw score by the total
number of potential D and E responses to determine the course value added rating. For example, let’ s
say 20 employees returned surveys for the course listed in Attachment 1. The evaluator assigns
numbers to the Column D and E task responses in which transfer of training occurred. The evaluator
adds up the numbers and determines that the raw score is 590. Then the evaluator divides the raw
score (590) by the total number of potential D and E responses (360) to determine the value-added
rating (1.64). Note that the highest possible value added rating is 5.0 (all training transferred,
employees use all of it daily on the job, and consider it all to be very important) and the lowest possible
rating is 0.0 (no transfer of training occurred at all).

The course value-added rating provided in the example listed above (1.64) is an actual one from a
transfer of training field test. Don’ be surprised if your course ratings are low: these are stringent
t
criteria. The value-added rating provides you with a unique means of quantifying the true value of
training. Not only does it factor in the transfer of training rate, but it also factors in training-acquired
skill usage, and task importance. As with the other measures, the most important thing that you can do
with the valued-added ratings for your courses is to compare them to one another. A course with a
comparatively high value-added rating is one to emulate. A course with a comparatively low value-
added rating is one to fix.

Finally, the value-added rating can be an important tool during efforts to convince your organization of
the value of training. Unlike the transfer of training rate, the value-added rating is something that
executives and line managers typically relate to and can appreciate. Remember that a high value added
rating means that employees learned, retained, and frequently use skills that are critical to the success
of the organization.

9. Calculate the transfer of training rate, prior knowledge rate, the “didn’ stick” rate, and the value added
t
rating for each course objective. Improve the course based on the results.

Reassemble all of the completed surveys. For each course objective (task in Column A), calculate the
transfer of training rate, prior knowledge rate, the “didn’ stick” rate, and the value added rating using
t
the basic methodology established in steps 5-8. For example, an evaluator is calculating the transfer of
training rate for Task #1 under Column A. Twenty employees returned course surveys. The evaluator
reviews Task #1 on each of the 20 surveys, marking those that match the training transfer pattern (B =

5

NO, C = YES, D = D, W, M, or Y). Eleven task responses match the training transfer pattern. The
evaluator divides the total number of Task #1 responses that match the training transfer pattern (11) by
the total number of item response (20) to calculate the transfer of training rate (55%) for Task #1. And
so on for each of the Trans-Measure rates.

The course objective rates provide the real meat for improving instruction. Most problems and
solutions will be self-evident. For example, in the previously mentioned business English field test, the
prior knowledge rate for an objective was nearly 90%. The evaluator noted that nearly all respondents
indicated that the task was not very important. The solution? Eliminate the objective and associated
content and re-assign the time to an objective that employees indicated was important, but had a high
“didn’ stick” rate. Using this methodology, the business English instructor was able to raise the
t
transfer of training rate for the course from 53% to 71%.

10. Distribute executive summary of survey results.

After you have calculated all of the Trans-Measure rates, develop an executive summary of the results
and distribute it to survey participants, training group personnel, line organization management, and
organizational executives as appropriate. The key here is to provide the information in a concise,
understandable fashion. Attachment 3 contains an executive summary for the course listed in
Attachment 1. This is only an example; use the distribution method and format that work best for your
organization.

11. Continue the measurement and improvement process.

Repeat this process for other courses during the six-month window. The more courses you evaluate,
the more data you will have to make meaningful decisions about ways to improve the transfer of and
the value added by your training courses.

6

Attachment 1 – Sample Survey

Dear Fellow Employee,

Time flies quickly . . . it has been six months since you took the Employee Selection Course for Managers. Please help the
organization improve the course by completing the following short questionnaire. We treat all individual input as confidential
information. After you complete the questionnaire please use the REPLY button to e-mail this back to me at [insert your e-mail
address]. Once we have compiled the results for this course we will send you a summary of course effectiveness, and what we intend
to do to improve the course. Please return your completed survey to me by July 20.

Thanks for your help,

Bill Keeley

Employee Selection Course for Managers – Taken January 1999

A B C D E

Scale
D = Daily
-- = Unimportant

interview

2 Evaluate employment Yes No Yes No D W M Y N -- - + ++ +++
application

3 Conduct selection tests Yes No Yes No D W M Y N -- - + ++ +++

4 Conduct employment Yes No Yes No D W M Y N -- - + ++ +++
interview

5 Check references Yes No Yes No D W M Y N -- - + ++ +++

6 Request a background Yes No Yes No D W M Y N -- - + ++ +++
check

-CONTINUED-

7

A B C D E

Scale
D = Daily
-- = Unimportant

7 Make selection Yes No Yes No D W M Y N -- - + ++ +++
decision

8 Make arrangements Yes No Yes No D W M Y N -- - + ++ +++
with HR to employ
individual

9 Notify candidates of Yes No Yes No D W M Y N -- - + ++ +++
decision

8

Attachment 2 – Blank Survey
Dear Fellow Employee,

Time flies quickly . . . it has been six months since you took the [insert course name]. Please help the organization improve the
course by completing the following short questionnaire. We treat all individual input as confidential information. After you complete
the questionnaire please use the REPLY button to e-mail this back to me at [insert your e-mail address]. Once we have compiled the
results for this course we will send you a summary of course effectiveness, and what we intend to do to improve the course. Please
return your completed survey to me by [insert date].

Thanks for your help,

[insert your name]
[Insert course name and date course was taken]

A B C D E

Scale
D = Daily
-- = Unimportant

1 [Insert objective] Yes No Yes No D W M Y N -- - + ++ +++










9

Attachment 3 – Sample Executive Summary
Dear Colleague,

As you know, we conduct a process to systematically measure and improve the effectiveness of our training. We surveyed the 83
employees who took the Employee Selection Course for Managers last January. Over 80% of course attendees returned surveys.
The following table shows the results of this survey, survey interpretation, and our plans to improve the course. Thank you to all for
making this improvement process a success.

Bill Keeley

Employee Selection Course for Managers – Course Taken in January 1999

Results Interpretation Improvement Plan

Measurement This Organizational
Course Average

Transfer of This indicates the amount of training By making the improvements listed
Training Rate 53% 41% that the trainees successfully applied on below, the training group is targeting a
the job. The course performed above transfer of training rate of 60% for the
our organizational average in this course sessions offered in November.
category.

Prior This indicates the amount of the Significantly reduce the course time
Knowledge 20% 34% training that trainees knew before spent on making arrangements with
Rate taking the training. The course HR to employee individuals. The
performed better than our survey indicates that over 80% of
organizational average in this category. trainees know how to do this.

“Didn’ Stick”
t This indicates the percentage of skills Increase the amount of time spent on
Rate 16% 28% taught in the course that trainees did selection tests, adding selection test
not learn how to perform and/or simulations. Managers don’ t
couldn’ remember how to perform.
t administer these tests often, but it is
The course performed better than our critical that they administer them
organizational average in this category. correctly when they do. The survey
indicates that 40% of trainees could
not perform this task today if asked.

Value-Added This indicates the amount of value the By making the improvements listed
Rating 2.55 1.23 training course added to the below, the value-added rating for this
organization on a 0 to 5 scale. A high course should improve. In addition,
rating indicates that employees are the training group will revise the
learning skills critical to the success of resume screening content to
the organization and applying them in emphasize how much important
their jobs. The course performed better information about individuals can be
than our organizational average in this extracted from their resumes.
category

10

Filename: TMeasure_w.doc
Directory: J:tech_shareTrans-Measure
Template: C:Program FilesMicrosoft OfficeTemplatesNormal.dot
Title: Dear [insert employee’ first name],
s
Subject:
Author: Bill Keeley
Keywords:
Comments:
Creation Date: 09/13/99 7:50 AM
Change Number: 3
Last Saved On: 09/13/99 8:59 AM
Last Saved By: Tony Alston
Total Editing Time: 2 Minutes
Last Printed On: 09/13/99 8:59 AM
As of Last Complete Printing
Number of Pages: 10
Number of Words: 3,468 (approx.)
Number of Characters: 19,773 (approx.)

Trans-Measure: Tool for measuring training transfer effectiveness

Recommended

Recommended

More Related Content

Similar to Trans-Measure: Tool for measuring training transfer effectiveness

Similar to Trans-Measure: Tool for measuring training transfer effectiveness (20)

Recently uploaded

Recently uploaded (20)

Trans-Measure: Tool for measuring training transfer effectiveness