1. .
THE TAXONOMY OF TRAINING AND DEVELOPMENT
OUTCOMES
(TOTADO)
EVALUATION FRAMEWORK
Dr Kamal Birdi
Institute of Work Psychology
Management School
University of Sheffield
Sheffield S10 2TN
United Kingdom
Email: K.Birdi@Sheffield.ac.uk
Tel: +44 (0)114 222 3288
This document contains a description of the TOTADO training and development
evaluation framework and two tools to help with its implementation and data
collection.
1. A paper summarising the TOTADO framework
2. A TOTADO evaluation strategy planning template
3. A TOTADO evaluation interview schedule
1
2. .
THE TAXONOMY OF TRAINING AND DEVELOPMENT OUTCOMES (TOTADO):
A NEW MODEL OF TRAINING EVALUATION
Kamal Birdi
Institute of Work Psychology, University of Sheffield
Paper presented at the Annual BPS Division of Occupational Psychology Conference, 13th-15th
January 2010, Brighton, UK.
INTRODUCTION
The effectiveness of training and development activities can be assessed by the extent to which those
activities produce desirable outcomes and reduce undesirable ones. Despite organisations expending a
great deal of effort setting up training and development programmes, comparatively less attention is
paid to systematically evaluating their effectiveness (Sadler-Smith, 2006). This lack of assessment
may be partly due to the extra cost and effort involved but it also seems that many Human Resource
practitioners are unaware of how and what to evaluate (Bates, 2004). The objectives of this paper are
therefore to briefly review and critique past models of evaluation and then offer a new model of
evaluation (the Taxonomy of Training and Development Outcomes – TOTADO) which addresses
these shortcomings.
A review of the literature indicates that three types of approaches can be described when considering
how outcome evaluation has been previously addressed. First, the predominant traditional training
evaluation approach (e.g. Kirkpatrick, 1959; Warr et al., 1970) offers a practical framework for
assessing outcomes in terms of trainee reactions, learning of knowledge and skills, on-the-job
behaviour and organisational performance. Although popular in organisations, these models tend to
be theoretically and practically vague in their specification of different types of learning outcomes,
work behaviours and organisational performance criteria (Phillips & Phillips, 2001). A second
approach is illustrated by Kraiger et al. (1993), who have proposed a classification scheme for
individual learning outcomes (cognitive, skill-based or affective) based on psychological theory. This
framework is useful for defining the specific types of criteria which indicate learning and it allows
theoretical rationales to be developed when choosing predictors of learning outcomes. However, this
second approach has rarely been used in subsequent research studies (Beech & Leather, 2006) and,
since it only focuses on learning outcomes, it also needs to be incorporated into the wider frameworks
offered by Kirkpatrick (1959) and Warr et al. (1970). Finally, the employee development literature
(e.g. Benson et al., 2004; Maurer et al., 2002) has empirically assessed experienced or perceived
development outcomes from different types of learning and development activities. The findings from
these studies indicates that such activities can lead to changes in a variety of areas not considered by
traditional training evaluation approaches, such as changing employees’ status, relationship to
colleagues or even health. A broader perspective on types of outcomes beyond individual learning of
knowledge and skills, work behaviour and organisational performance is therefore needed if training
and development effectiveness is to be accurately assessed. The Taxonomy of Training and
Development Outcomes (TOTADO) attempts to do this (see Figure 1) and will be outlined.
2
3. .
DESCRIPTION OF THE TAXONOMY OF TRAINING AND DEVELOPMENT OUTCOMES
TOTADO integrates and elaborates on previous evaluation approaches to define a taxonomy of
outcomes that allows us to capture and describe the diverse range of outcomes experienced from
participation in both training and other development activities. The research described earlier revealed
that although the impact of development activities can be assessed at differing levels, much of the
work has concentrated on the individual. Kirkpatrick’s model of evaluation has been mainly used to
consider the individual level of analysis from Level 1 to Level 3; Kraiger et al.’s (1993) taxonomy
considers solely individual level learning outcomes; most studies from the employee development
literature have used surveys to assess the impact of training and development activities on their
participants. Consequently, the outcomes described in the previous section refer predominantly to the
individual, whereas other levels of outcomes are equally important. Within the TOTADO framework,
it is therefore said that outcomes can be measured at four basic levels: individual, team (or work
group), organisational and societal.
Figure 1. The Taxonomy of Training and Development Outcomes (TOTADO)
Framework.
1. Individual-level outcomes
3
4. .
With TOTADO we extend previous evaluation models by elaborating five different types of
individual-level outcomes where the trainee is the unit of analysis: Affective (e.g. participant reactions
to training, motivation, self-efficacy, attitudes, mental well-being); Cognitive (verbal knowledge,
knowledge representation); Behavioural (off-the job task performance, on-the-job task performance);
Physical (e.g. health, fitness, injuries): and Instrumental (events, actions or status changes resulting
from participation in training and development activities that are work intrinsic (increased job control),
work extrinsic (pay rise, promotion) or work relational (forming new relationships or networks)).
2. Team-level outcomes
With the increasing implementation of teamworking in the workplace, there has been a concurrent rise
in team training and development activities (Wilson et al., 2007). For example, many organisations
send work groups on external “outward-bound” courses where team members are required to co-operate
and work closely to achieve given objectives. Evidence to date of the impact of such activities
tends to be anecdotal, unsystematic and, in particular, ignores whether there is any change in team
functioning back in the workplace (Salas et al., 2004). Hence, we need to consider using the team or
work group as the unit of analysis for assessing the impact of these types of opportunities. The
majority of the outcomes which will be described for individuals can also be applied by aggregation to
the team context, although there is more focus on measuring interpersonal behaviours and group
cognition and affect. The four dimensions of impact here where the team or group is the unit of
analysis are therefore: Affective (e.g. changes in average team identity or trust); Cognitive (verbal
knowledge, knowledge representation in terms of shared cognition); Behavioural (team task processes,
intra-team processes); and Instrumental (events, actions or status changes for the team as a whole
such as increased team autonomy or gaining team bonuses).
3. Organisational-level outcomes
The next level takes the unit of analysis to be the organisation, or major functional components of it
(e.g. departments) and outcomes relate to those directly involved with the organization in a
transactional format (e.g. employees, customers, shareholders). Although individual- or team-level
outcomes can be aggregated to provide organisational-level outcomes (e.g. proportion of company
employees with a certain technical qualification), some variables can only be measured at the
organisational level (e.g. company profitability, share values). This is the domain that Kirkpatrick
(1959) refers to as Level 4 or Business Results, although he remains vague in terms of what aspects
should be included in this level. It would be conceptually very helpful to identify key categories into
which we can place organisational-level outcomes, much as was done with the individual and team
levels. The organisational effectiveness literature highlights the complex nature of measuring
performance but does attempt to provide some broad criteria. We therefore draw on the work of
management researchers (Cameron, 1980; Kaplan & Norton, 1996; and Richard et al., 2009) to specify
four dimensions of organisational performance outcomes: Financial (e.g. turnover, profit, share price);
Outputs (quantity, quality, variety of components, products or services); Processes (how well the
4
5. .
organisation functions e.g. time to complete tasks, stoppages, communication system efficiency); and
Resources (human and non-human).
4. Societal-level outcomes
The final level is rarely considered but relates to the impact of training beyond the organisational level
where the area or group outside the organisation is the unit of analysis (Watkins et al, 1998). The
majority of past models focus solely on impacts within the organisation, yet a wider perspective is
sometimes needed. For example, training employees in carbon-friendly practices within a
manufacturing company may translate to decreased waste and pollution levels for surrounding
communities. At this level, we can therefore also assess the impact of training on a sectoral, regional
or even national bases. For instance, does training all Job Centre Plus staff in coaching techniques to
help the long-term unemployed significantly improve regional employment rates? The five broad
dimensions of societal impact we can consider are therefore: Economic (e.g. £ investment in region);
Health and Welfare (e.g. mortality rates from heart disease); Educational (e.g. level of qualifications of
local populations); Law and Order (e.g. local crime rates): and Environmental (e.g. pollution levels).
DISCUSSION
The TOTADO framework therefore presents an approach which integrates and enhances past models
of evaluation by presenting multiple levels of impact and specifying within each level the
key dimensions of outcome change. Key features worth noting include:
· It acknowledges that the effectiveness of training and development participation should be
assessed on a number of levels and using a range of relevant criteria within each level. Conceptually,
it integrates the focused view of the traditional Kirkpatrick training evaluation approach with the wider
evaluation perspective offered by other literatures. By using the taxonomy in the early stages of
developing an evaluation strategy, researchers and practitioners can become fully aware of the variety
of salient outcome criteria that may need to be measured.
· The taxonomy clearly distinguishes between individual, team, organisational and societal
measures of organisational effectiveness. Approaches such as Kirkpatrick’s have tended to remain
vague about measures of individual work behaviour and organisational results and seemed to have
ignored team-level analysis altogether.
· The nature of the taxonomy also allows detailed and comparative examination of the types of
outcomes produced by different learning activities. For instance, how much impact does participation
in formal training compared to work-based development activities have on employees’ general job
satisfaction? In situations where several studies have examined the same development activity but
assessed different types of outcomes, the taxonomy provides a framework to integrate the findings in
order to provide a more complete overview of the impact of that activity. It would also highlight
which types of outcomes had not yet been assessed and hence drive future research to address those
gaps.
5
6. .
· Since the framework is a taxonomy, it does not make any claims of causality between outcomes.
Although Kirkpatrick (1996) claims that his framework is solely a taxonomy, it has widely been
interpreted as a training effectiveness model which specifies causal linkages between individual and
organisational outcomes (Alliger et al., 1997). If we are truly to identify the factors that influence
training outcomes, just considering other outcomes is clearly insufficient. Previous research has
consistently shown that a wide range of individual, training-related and organisational factors can also
have an impact on different types of outcomes (Noe, 2008). Such factors need to be included in any
good training effectiveness model. The evaluation taxonomy presented here, however, does provide
the building blocks for creating models which propose causal linkages between different types of
outcomes and other types of variables.
· Finally, the evaluation taxonomy provides a common format for assessing learning and
development impact from a diverse range of theoretical approaches. For example, cognitive scientists
conducting laboratory studies on learning complex tasks can be regarded as examining learning
effectiveness in terms of individual-level cognitive or behavioural outcomes that are measured in an
off-the-job context. Field studies conducted by occupational psychologists examining the transfer of
training to the workplace may focus on individual work behavioural outcomes. Team researchers can
be considered as assessing the impact of development activities in terms of changes in team
behaviours such as communication and co-operation. Organisational researchers go beyond individual
and team levels and relate the amount of training offered by companies to organisational-level goal-directed
outcomes such as sales or turnover (Wright et al., 2005). The framework should therefore be
flexible enough to cater for a wide variety of paradigms where learning effectiveness is to be assessed.
The presentation will illustrate the utility of TOTADO by providing examples from research
conducted by the author including:
· An evaluation study which shows how three types of creativity training programme run within a
government department can generate different levels of individual-level affective, cognitive and
behavioural outcomes for participants.
· A study of 500 UK organisations showing the relationship between organisations’ use of
individual training and development practices and different organisational performance outcomes.
REFERENCES
Alliger, G. et al. (1997). A meta-analysis of the relations among training criteria. Personnel
Psychology, 50, 341-358.
Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the
principle of beneficence. Evaluation and Program Planning, 27, 371-347.
6
7. .
Beech, B., & Leather, P. (2006). Workplace violence in the health care sector: A review of
staff training and integration of training evaluation models. Aggression and Violent
Behavior, 11(1), 27-43.
Benson, G. S., et al. (2004). You paid for the skills, now keep them: Tuition reimbursement
and voluntary turnover. Academy of Management Journal, 47(3), 315-331.
Cameron, K. (1980). Critical questions in assessing organizational effectiveness.
Organizational Dynamics, Autumn, 66-80.
Kaplan, R., & Norton, D. (1996). Linking the balanced scorecard to strategy. California
Management Review, 39, 53-79.
Kirkpatrick, D. (1959). Techniques for evaluating training programs. Journal for the
American Society for Training and Development, 13, 3-9.
Kraiger, K., et al. (1993). Application of cognitive, skill-based and affective theories of
learning outcomes to new methods of training evaluation. Journal of Applied
Psychology, 78(2), 311-328.
Maurer, T. e al. (2002). Perceived beneficiary of employee development activity:A three-dimensional
social exchange model. . Academy of Management Review, 27(3), 432-
444.
Noe, R. A. (2008). Employee Training and Development (Fourth Edition). New York:
McGraw-Hill.
Phillips, P. & Phillips, J. (2001). Symposium on the Evaluation of Training: Editorial.
International Journal of Training and Development, 5(4), 240-247.
Richard, P. et al. (2009). Measuring Organizational Performance: Towards Methodological
Best Practice. Journal of Management, 35(3), 718-804.
Sadler-Smith, E. (2006). Learning and Development for Managers. Oxford: Blackwell
Publishing.
Salas, E. et al. (2004). 25 years of team effectiveness in organizations: Research themes and
emerging needs. In C. Cooper & I. Robertson (Eds.), International Review of
Industrial and Organizational Psychology (Vol. 19, pp. 47-91). Chichester: John
Wiley & Sons.
Warr, P. et al. (1970). Evaluation of management training. London: Gower Press.
Watkins, R et al. (1998). Kirkpatrick plus: Evaluation and continuous improvement with a
community focus. Educational Technology, Research & Development, 46(4), 90-95.
Wilson, J. et al (2007). Group Learning. Academy of Management Review, 32(4), 1041-1059.
Wright, P. et al. (2005). The relationship between HR practices and firm performance:
Examining causal order. Personnel Psychology, 58, 409-446.
7
8. .
TOTADO EVALUATION STRATEGY FRAMEWORK
NAME OF TRAINING / DEVELOPMENT PROGRAMME:
Level Dimension of
effectiveness
Type of information Methods of collecting
information
Who provides
the information
When to collect information
TIME 0 TIME 1 TIME 2 TIME 3
period before
at the start of
at the end of
the course
the course
the course
period after
the course
INDIVIDUAL
Affective Reactions, attitudes, motivation,
self-efficacy, mental well-being
Cognitive Verbal Knowledge,Knowledge
Representation
Physical Health, Fitness, Injuries
Behavioural Off-the-job, on-the-job task
performance
Instrumental Work intrinsic, work extrinsic,
work relational, nonwork
TEAM
Affective Shared team attitudes
Cognitive Shared cognition
Behavioural Team performance, team
interaction processes
Instrumental Work intrinsic, work extrinsic,
work relational,
8
10. .
TOTADO INTERVIEW SCHEDULE
“In the following interview, I shall be asking you to describe any changes that you think have
arisen because of programme X.”
A. INDIVIDUAL OUTCOMES
GENERAL LEAD IN
As a result of taking part in X, how do you think you have changed?
AFFECTIVE
How has taking part in the X changed the way you FEEL about things?
COGNITIVE
What type of things have you LEARNED as a result of taking part in X?
PHYSICAL
How has taking part in the X changed your LEVELS OF HEALTH AND FITNESS?
BEHAVIOURAL
What can you DO now that you couldn’t do before taking part in X?
How has taking part in X changed your JOB PERFORMANCE?
INSTRUMENTAL
Has taking part in X led to any changes in the type of work you do?
Has taking part in X led to any changes in your job status or conditions of employment?
Has taking part in X helped you form new relationships at work or changed existing
relationships?
Has taking part in X led to any other changes in your work that we haven’t discussed yet?
Has taking part in X led to any changes for you outside work?
IDENTIFYING BARRIERS TO CHANGE IN INDIVIDUAL OUTCOMES
Has anything in particular stopped or hindered you changing personally the way you wanted
after the training / taking part in Programme X?
Is there anything that could be changed about Programme X, your organisation or other
aspects that could help you achieve these desired changes?
10
11. .
B. TEAM / GROUP OUTCOMES
GENERAL LEAD IN
As a result of taking part in the X programme, how do you think your team has changed?
AFFECTIVE
How has taking part in the X changed the way your team FEELS about things?
COGNITIVE
What type of things has your team LEARNED as a result of taking part in X?
BEHAVIOURAL
What can your team DO now that you couldn’t do before taking part in X?
How has taking part in X changed your team’s WORK PERFORMANCE?
INSTRUMENTAL
Has taking part in X led to any changes in the type of work your team does?
Has taking part in X led to any other changes in the type of work your team does?
Has taking part in X helped your team form new relationships at work or changed existing
relationships?
Has taking part in X led to any other changes in your team’s work that we haven’t discussed
yet?
IDENTIFYING BARRIERS TO CHANGE IN TEAM/GROUP OUTCOMES
Has anything in particular stopped or hindered your team or teams changing the way that was
wanted after the training / taking part in Programme X?
Is there anything that could be changed about Programme X, your organisation or other
aspects that could help you achieve these desired changes?
11
12. .
C. ORGANISATIONAL OUTCOMES
GENERAL LEAD IN
As a result of running the X programme, how do you think your organisation has changed?
OUTPUTS
Has running X led to any changes in your organisation’s outputs e.g. its productivity, quality
levels, variety of products/sevices offered etc.?
RESOURCES
Has running X led to any changes in your organisation’s employees e.g. their performance,
morale, absenteeism?
Has running X led to any changes in your organisation’s material resources e.g. reductions in
wastage?
PROCESSES / OPERATIONS
Has running X led to any changes in your organisation’s efficiency or the way work is carried
out e.g. speed of production, turnaround on project times?
FINANCIAL
Has running X led to any changes in your organisation’s financial performance e.g. its
profitability, balancing of budgets?
Has taking part in X led to any other changes in your organisation that we haven’t discussed
yet?
IDENTIFYING BARRIERS TO CHANGE IN ORGANISATIONAL OUTCOMES
Has anything in particular stopped your department or organisation changing the way that was
wanted after the training / taking part in Programme X?
Is there anything that could be changed about Programme X, your organisation or other
aspects that could help you achieve these desired changes?
12
13. .
D. SOCIETAL OUTCOMES
GENERAL LEAD IN
What types of general impact would you say the X programme has had on communities or
groups outside the organisation?
ECONOMIC
What types of general impact would you say the X programme has had on the economy in
your region ?
e.g. £ investment in geographic region, unemployment rates in area
HEALTH AND WELFARE
What types of general impact would you say the X programme has had on the health or
welfare of the community in your region?
e.g. no. of deaths from heart disease, alcohol-related illnesses in local population
EDUCATIONAL
What types of general impact would you say the X programme has had on the education
levels of the community in your region ?
e.g. level of qualifications of local population, % social inclusion of minorities
LAW AND ORDER
What types of general impact would you say the X programme has had on law and order
issues in the community in your region?
e.g. no. of robberies in the area, % reduction in drug crime
ENVIRONMENTAL
What types of general impact would you say the X programme has had on the geographical
environment of your region?
e.g. pollution and waste levels in the region
Has taking part in X had any other impacts on the local or national community that we haven’t
discussed yet?
IDENTIFYING BARRIERS TO CHANGE IN SOCIETAL OUTCOMES
Has anything in particular stopped or hindered your organisation’s impact on the community
or society in the way you wanted after the training / taking part in Programme X?
Is there anything that could be changed about Programme X, your organisation or other
aspects that could help you achieve these desired changes?
13
