1. Towards Managing the Learning Process: Evaluation of
Simulation Training
Rebecca J. Atkins1; Malcolm R. Smith1; Anthony R. Mildred2; & H. Peter Pfister1
1
School of Behavioural Sciences, The University of Newcastle
2
NSW RailCorp
Rebecca.Atkins@newcastle.edu.au
Abstract. The progressive nature of the simulation industry requires advanced training techniques, the strategic core
of which is the management of learning. A systematic approach to learning is further reliant on the development and
implementation of effective evaluation strategies. Leaders in the field have identified the need for the re-development
of evaluation methods including outcome measures, system assessment, and the widespread acceptance and
incorporation of evaluation techniques. The current need for training evaluation extends beyond the well established
validation of simulators as training tools. Effective evaluation structures increase the effective use of training systems
and the application of skills acquired during the training program, further enabling the scientific validation of all facets
of the training process. The development of effective training evaluation processes grounded in psychometrics and
cognitive science is essential for the continuation of advanced training within the simulation community. This paper
presents an external evaluation of an industry developed evaluation process of a reality training centre utilising
immersive simulation within the rail transport industry. The evaluation related to a recent round of a safety
management system training program delivered across all levels of the rail workforce within an Australian State Rail
Corporation. The measure was not found to provide an accurate assessment of identified program objectives.
However, valuable insights into workforce perceptions of the training process, programs, and facilities were
established. Discussion highlights the identification and evaluation of the learning components of the training task, the
effective and efficient use of the training facility, the enhancement of the training process, and the re-development of
an effective training evaluation system. Evaluation essentially represents the ability to understand the multifaceted
training process and the learning outcomes produced, allowing application of this knowledge for further advancement
of simulation use and effective training techniques.
within the field of simulation training and the limited
1. INTRODUCTION acceptance and incorporation of evaluation techniques
The progressive nature of the simulation industry is requires persistent intervention [6],[8]. The need for the
reliant upon advanced training techniques centring on re-development of evaluation strategies has been
the cognitive concepts of human learning. identified by industry personnel, highlighted as a means
Internationally, the emerging role of the behavioural of continued advancement [6]. This need extends
sciences and the increasing presence of Human Factors beyond assessment of technology and the well
(HF) within the cross-disciplinary industry of simulation established validation of simulators as training tools [9],
are reenergising the pursuit of advancement through a [10] to psychometrics and cognitive science with the
focus on human and system performance [1],[2],[3]. The ongoing evaluation of all facets of the training process
strategic core of this approach is the management of with particular emphasis on the role of learning
learning. Research has identified the progressive need [11],[12],[13]. Evaluation is an essential adjunct to
within the simulation industry to extend training simulation training aimed at optimising the effective use
practices beyond the current technological focus to of technology and maximising performance outcomes.
intensify the learning aspect of training [4],[5]. Such
emphasis serves to maximise the use of simulators as 1.1 Training and Evaluation
training tools [6] and to further advance the industry and Training is the systematic process of instruction,
the performance outcomes of its professionals. practice, review, and examination [14]. Evaluation is an
Psychologists within the simulation industry and within integral part of this process, offering a strategic
the current context of rail transportation aim to develop approach to advancement [11]. Evaluation is the process
methods by which systems can be improved to better of gathering information to ascertain the effectiveness
accommodate the cognitive, physiological, and and efficiency of training programs to make informed
behavioural limits of human operators to maximise the improvements to training [14]. Effective evaluation
effectiveness of training tools and minimise frequency structures increase the effective use of training systems
of human errors and resulting accidents [7]. The and the application of skills acquired during the training
investigation and strategic improvement of training program. It is suggested that in general industry very
methods is a cost-effective means of improving human few training programs are effectively evaluated and
performance within a system and improving the subsequently improved upon [11],[14]. Empirical
functionality of the system itself: a process referred to as evidence indicates evaluation is the most poorly
evaluation. The current lack of psychometric evaluation performed element of training and is often not

2. performed at all [8],[11],[15]. Early theories of training physiological and behavioural limits of human operators
present evaluation as an essential and final phase of the maximising efficiency and effectiveness and improving
training process that serves as the measure from which safety [3].
learning is examined and training effectiveness derived
[12]. Kirkpatrick’s [12] model of training evaluation
remains dominant within industrial/organisational
psychology [8],[15]. The model presents four levels of
training evaluation, each increasing in depth. The four
levels (presented in Figure 1) are: (1) how trainees feel
about the program, (2) quantity of learning in the form
of increased knowledge and understanding, (3) changes
in behaviour, and (4) effects of behavioural change on
the attainment of learning objectives. These can be Figure 2: The ‘training cycle’ model [11].
further simplified as: (1) reactions and feelings, (2)
learning, (3) behaviour, and (4) results. The higher
levels of evaluation refer to changes in learning and 1.2 Training and Evaluation in Rail Transportation
behaviour where learning is represented by the Human error is well documented as the leading
acquisition of skills and knowledge and adoption of new contributor to in excess of 80% of all transportation
behaviours [16]. The attainment of desired learning accidents [1],[5],[19]. HF investigations into complex
outcomes is vital to the effectiveness of training modern transport systems have revealed the ability for
[8],[11],[13],[17]. The Kirkpatrick model highlights the dramatic reduction of error rates through effective
fundamental aim of training programs to achieve change training [1],[19]. Research has further shown effective
and/or development in the cognitive knowledge, skills, training to be a cost-effective method for the reduction
and/or attitudes of trainees indicating the achievement of HF related transportation accidents [4]. Accurate
of learning outcomes [18]. Essentially training statements as to training effectiveness within the
effectiveness is determined by measuring how well Australian rail industry cannot be made to date, as no
programs meet learning outcomes set for training and publicly available/published scientific evaluation has
ultimately the level of behavioural transfer into the taken place. Without an effective evaluation process
workplace [11],[13],[18]. there is no evidence to suggest the presented concepts or
learning objectives of a training program will be
transferred into behaviour [11]. This is cause for
concern particularly when training is used as an
organisational strategy for the reduction of human error
as a causal factor of accidents, as is the case with
immersive simulation (IS) training within the Australian
rail transport industry. The unquestionable importance
of evaluation was stressed in the inquiry into the
Glenbrook rail disaster. Employees who participated in
safety training (which has not been evaluated) failed to
exhibit key cognitive reasoning skills resulting in failure
Figure 1: Four level model of evaluation [12]. to take appropriate action to avert the collision. The
inquiry findings suggest effective evaluation of the
A best practice model of training has been proposed by
training program may have averted the disaster [20].
Bramley [11] extending the Kirkpatrick model and
Developing effective measurement tools to evaluate rail
emphasising the integral role of evaluation in the wider
safety training is paramount and is a strong contributor
training process. According to the ‘training cycle’ model
to the rationale of the current study.
[11], the design of training involves the ordered process
of: (1) identification of training needs, (2) setting
learning objectives, (3) selection of appropriate training 1.3 Background and Rationale for the Research
methods, (4) delivery of the program, and (5) evaluation The use of IS as a key element of safety training is
of the program and the feedback of information. relatively new to the Australian rail transportation
Integration of each of these steps is depicted in Figure 2. industry, with the Australian Rail Training Reality
The model indicates that information derived through Centre (ARTRC) completing the first round of training,
training evaluation (in particular higher level Safety Management System 2.1 (SMS 2.1), in 2001.
evaluation), has the potential to influence training Prior to this, interactive operator simulation was used
programs, possible safety issues, required competencies, throughout the early 1990s to improve operator
and the way work roles and organisational systems are performance in freight rail transportation and interactive
defined. This has specific implications for the passenger train simulation was incorporated into driver
development of systems that control for human error training programs in the mid 1990s. No scientific
and other HF considerations. Information gained investigation has been reported regarding the
through the evaluation of training has the potential to effectiveness of these programs. The IS currently
alter the design of the workplace to better accommodate utilised within the ARTRC SMS program offers non-

3. interactive group training involving scenario/incident the questionnaire is not based on the learning outcomes
simulations using the immersive audio-visual capacity set for the training program and is limited to evaluating
of the facility. The non-interactive basis of the the IS experience. Furthermore, the questionnaire was
simulation permits training (and associated benefits) to not based on any previously validated evaluation tool(s).
extend to trainees in both driving and support roles as Hence, the validity of the SMS 2.4 EQ is not known,
technical skills are not required to complete the training. and no previous testing or development of the
Such an approach is aligned with program objectives to instrument has taken place. The SMS 2.4 EQ consists of
enhance cohesion and communication among the 18 questions printed on a single sided A4 sheet in eight
workforce. SMS training is further conducted with point font. The first series of questions relate to
mixed workgroups aimed at developing teamwork and a demographic information ascertaining identification,
positive safety culture through increasing the level of age, gender, workgroup, length and type of service. The
understanding of how different work roles are integrated remaining questions hold a more formalised (while
into the management of safe rail operations. The varied) response structure being either a forced-choice
adoption of this technique in interactive simulation categorical selection or a mixture of four- or five-point
training has successfully enhanced teamwork between Likert response scales. However, the Likert scale
work roles in both aviation and medicine [5],[21]. The response options are not standardised between
mixed group non-interactive IS training used at the questions, varying according to item topics.
ARTRC is however, a unique concept that is previously Questionnaire items refer to impressions of mixed group
untested as a learning tool. If such SMS training is to be training, perceptions of the IS scenarios in both SMS
reliable the quality and effectiveness of the training must 2.4 and previous training rounds, increased awareness of
be ascertained. safety defences, and impressions of learning.
1.4 Current Study 2.3 Procedure
The aims and objectives of the current study were set by No manipulation or control of the IS process was
ARTRC executive (the project body) to determine the conducted as part of the current experiment. The
outcomes of IS training and the mixed group training procedure was limited to analysis of the existing SMS
technique. The specified objectives of the program are 2.4 EQ dataset. The original dataset was collected by
that IS SMS training: (1) increases understanding, (2) the program instructors at the completion of SMS 2.4
increases learning, (3) increases awareness of individual training. No independent data collection was conducted
responsibility for safety, (4) promotes discussion about as part of this phase of the research.
safety, and (5) provides a positive response to mixed
group training. The current research examined an 2.4 Data Analysis
existing dataset from round 2.4 of the SMS program
(SMS 2.4). Analysis was conducted to establish the The dataset was analysed to evaluate the SMS 2.4 EQ.
constructs of the training evaluation measurement tool The original Microsoft Access database was transferred
(the SMS 2.4 Evaluation Questionnaire [SMS 2.4 EQ]) into Microsoft Excel and recoded into a numerical form
developed and implemented by the project body. The that could be analysed in SPSS version 12.01. Seventy-
SMS 2.4 EQ has not previously been empirically tested. one records were lost during the computer import/export
The purpose of the research was to examine the process as a result of data corruption. Standard quality
evaluation tool and develop recommendations for the control processes were conducted on the recoded dataset
optimisation of the evaluation process. to confirm accuracy.
2. METHOD 3. RESULTS
2.1 Participants 3.1 SMS 2.4 EQ Evaluation
2,427 employees across the rail workforce participated An Exploratory Factor Analysis was conducted on the
in SMS 2.4 as a required part of their employment. The dataset to identify the constructs measured by the SMS
sample consisted of 463 females, 1,947 males, and 17 2.4 EQ and determine whether the evaluation tool would
who did not specify gender. Fourteen represented generate a factor structure that corresponded with the
workgroups were simplified to: driver, station staff, objectives of the IS training. Principal Component
guard, network ops, and other. Both males and females Analysis (PCA) using Varimax rotation was used to
were represented across each of the workgroups and an isolate the factors measured by the questionnaire items.
even spread of age, length of service, and experience in The SMS 2.4 EQ yielded a 4-factor structure explaining
multiple work roles was recorded. 66.45% of the variance. The four factors identified were
as follows: Factor 1 – training systems and processes (7
items; Cronbach's alpha [CA] = 0·87); Factor 2 –
2.2 Materials previous IS training (2 items; CA = 0·76); Factor 3 –
The evaluation measure, the SMS 2.4 EQ, was current IS training (1 item); and Factor 4 – training in
developed by the project body as a measurement tool to mixed groups (1 item). From a technical perspective, the
evaluate the SMS 2.4 IS training program. The design of questions loading on Factor 1 are satisfactory while the

4. data for the remaining three Factors is at best suspect. on understanding SMS, coinciding with Factors 2 and 3.
Satisfactory CAs (a form of quality assurance) were Several questionnaire items specifically probed
noted for Factors 1 and 2. However, for Factors 3 and 4 participant perceptions, opinions, and views toward
comparative CAs could not be calculated as only one previous and current IS training. Within the SMS 2.4
item loaded on each, while the single item loadings on EQ design, response scales for these Factors differed.
Factor 3 (alpha = .86) and Factor 4 (alpha = .88) were To allow for statistical comparison, item responses were
strong. Construction of the questionnaire made it recoded into a two-point scale of either a positive or
difficult to compare items. Several questions require the negative response. Results indicate a positivity rating of
choice of one of three options that are unique 75% towards previous IS scenarios (Factor 2) and 85%
categorical responses to the particular question, toward SMS 2.4 (current) scenarios (Factor 3).
resulting in a lack of comparative power. A standardised Suggesting trainees on average found the IS experience
method of item responses is recommended for future assisted their conceptual understanding and also that the
questionnaires. Further PCA examination revealed many scenarios are improving from previous rounds of
questions loaded on more than one Factor indicating training. CSA revealed significant differences between
these items require revision to be more specific in order workgroups in terms of positive response to IS training
to provide meaningful information. Analysis of the SMS for Factor 2 (2 items) and Factor 3 (1 item): Factor 2 –
2.4 EQ suggests, in its current form, the evaluation tool item 1: ²(4, N = 2,226) = 54.17, p < .001; drivers
χ
is unable to measure training outcomes as intended. reported the most assistance from previous IS training
Examination reveals the questionnaire to be an emotive/ (83%), this result may have been contributed to by the
reaction and perceptual measure of the overall training corresponding scenario specifically relating to their
process measuring positivity toward training. However, work role, depicting a driver as the main character.
employee reactions to the program provide useful Factor 2 – item 2: ²(4, N = 2,311) = 37.34, p < .001;
χ
information for possible improvement of SMS training. station staff reported the most assistance from previous
Such information was derived through analysis of the IS training (80%), a result again potentially influenced
data relating to each of the four Factors. by the referenced scenario centrally depicting and
relating to station staff. Factor 3 – item 1: ²(4, N =
χ
3.2 Factor 1: Training Systems and Processes 2,427) = 36.27, p < .001; guards reported the most
assistance from current IS training (89%), interestingly
While the majority of items loading onto Factor 1 were where a guard was depicted as the main character of the
intended to measure different aspects of the outcomes of scenario. Such consistent findings strongly suggest
the training experience, they are in fact measuring the impressions of the IS training experience are closely
same construct identified as a measure of overall linked to trainees identifying with their work role in the
training systems and processes. Investigation of the scenario, further suggesting SMS is not meeting the
results of these items presents a positive view of the identified objective of enhanced teamwork through
overall training program with a positivity high of 84% increased understanding of different work roles.
reported by participants, providing encouraging However, the SMS 2.4 EQ revealed that experiencing
information from trainees about the use of the training the IS scenarios has a positive effect on the development
facility and the IS program. Demographic variables of understanding SMS for individual work roles.
were examined for Factor 1 responses using Chi square Workers when seeing a co-worker of the same status in
analysis (CSA). A significant difference in percentage of the simulation scenario may have found increased
positive responses towards the training systems and affinity toward the character and the scenario in general.
processes was revealed between workgroups ²(16, N =
χ
This aspect should be considered as a valuable guide in
2343) = 64.40, p < .001, where drivers (91%) had the the construction of future training scenarios.
highest rate of overall positive responses to SMS 2.4.
Such a finding contrasts previous anecdotal evidence
suggesting drivers disliked training. CSA also revealed a 3.4 Factor 4: Training in Mixed Groups
significant difference between males and females on Positive response to mixed group training within the IS
positive responses ²(4, N = 2333) = 187.44, p <.001.
χ
environment is an objective of the SMS training
Females consistently reported a lower rate of positivity program. The fourth Factor measured by the SMS 2.4
(64%) compared to males (89%). A question measuring EQ specifically relates to mixed group training.
the perceived degree of improved learning from the IS Analysis was conducted to determine whether training in
experience yielded a positive response from only 49% mixed groups had any effect on the perception of the
of females (compared to 84% of males), suggesting the training experience. Results indicate mixed group
current IS training program is not positively accepted training was overall not received well however, this may
among the majority of female trainees. Such a result have been confounded by the limited data available on
may have been contributed to by all the employee this construct (1 item). Overall 47% of participants
characters depicted in the training scenarios being male. responded positively to the mixed workgroup training
and 49% of participants responded negatively of which
3.3 Factors 2 and 3: IS Training 46% stated that mixed group training took time away
from gaining skills specific to their role and 3% reported
An objective of the training program was that mixed group training was not useful at all. CSA
experiencing the IS scenarios will have a positive effect revealed a significant difference in positive responses to

5. mixed group training for different ages ²(6, N = 2,342)
χ difference does exist in degree of positivity between
= 68.31, p < .001; trainees aged 36-45 years responded males and females. Initiating changes to future training
least positively (40%), younger workers may have found programs to eliminate gender bias in the simulated
the interaction with older, more experienced workers scenarios may improve the positivity response in
beneficial. CSA also revealed a significant difference females. The results also suggest a lack of understanding
between workgroups in positivity of response ²(8, N =
χ of teamwork between roles within the rail system with
2,343) = 29.87, p < .001; drivers were the least positive an identifiable preference for training within isolated
workgroup toward the mixed group training experience workgroups and a negative response for training in
(46%), combining this evidence with the response of mixed workgroups. Such a finding holds potential
drivers to Factor 2 item 1 suggests drivers prefer more negative influence on applied safety behaviour [5]. The
specific training within their workgroup. CSA further identification of predetermined learning outcomes
revealed a significant difference in positivity of would be the most reliable dependent variable to use in
response according to length of service ²(8, N = 2,343)
χ future research and development of the SMS 2.4 EQ. A
= 39.56, p < .001; trainees with 5-20 years of experience valid learning outcome evaluation tool would need to be
were least positive toward mixed group training (42%). developed as part of future research into the learning
Workers with either a shorter (< 5 years) or longer (> 20 benefits of the IS SMS program.
years) length of service responded more positively,
interaction with employees of contrasting experience 4.2 Recommendations
levels may have been perceived as beneficial. Overall,
the results derived from Factor 4 indicate the mixed In order to improve the current questionnaire the
group aspect of the training program is an area which following recommendations should be considered: (1)
can be improved to promote teamwork and positive Improved construction of the EQ including changes in
safety behaviour. the layout such as allowing more space between
questions, incorporating sub-sections, and using larger
font. (2) It is suggested that a standardised Likert scale
4. DISCUSSION be used in future EQs to allow for the expression of a
magnitude of responses and ease of item comparison.
4.1 SMS 2.4 EQ Effectiveness The use of forced choice categorical responses
The SMS 2.4 EQ was developed to evaluate the unnecessarily limits the information participants are able
objectives of the IS component of the SMS 2.4 training to express and also leads responses. Inclusion of open-
program. The derived factor structure of the evaluation ended questions would allow for greater transfer of
questionnaire (EQ) failed to support this supposition. employee feedback and not limit responses, further
Essentially the SMS 2.4 EQ measures the main factor of leading to identification of undiscovered concerns or
the perception and impression of training systems and information about the IS program. (3) Measuring
processes. It is concluded the SMS 2.4 EQ does not multiple constructs with long questions creates
validly measure the constructs that were specified as participant confusion, a lack of clarity in the
objectives for the IS program. The tool requires measurement tool, and results in items that are not pure
scientific development if a valid measure for evaluating measures. Short, concise questions would correct this
the program is to be obtained. The SMS 2.4 EQ is likely issue in future tools. The addition of multiple questions
an emotive/reaction and perceptual measure towards the based on predetermined constructs would lead to a more
overall training process measuring training positivity. reliable factor structure. Using negatively and positively
The SMS 2.4 EQ, in its current form, is a tool for phrased questions would also control for a positivity
conducting the first level of evaluation (see Figure 1). bias. (4) Fatigue may have been a confounding variable
Future directions based on the best-practice model of in the evaluation process. The SMS 2.4 EQ was
evaluation suggested by Kirkpatrick [12] should aim at completed at the end of a lengthy training day. A
increasing the depth of the EQ to a measure of higher possible alternate explanation of the results is that
order learning and behavioural change. The Factor 1 might actually be employee positivity towards
incorporation of effective evaluation into the training finishing the day and hence not positivity towards the
process as suggested by Bramley [11] would allow for training process. To minimise the likelihood of this
the optimisation of the safety training process and the IS confound, it is recommended future studies counter-
program. Development of evaluation tools exploring balance between training groups the time of the IS
multidimensional constructs based on declarative experience and related evaluation during the training
knowledge, cognitive and behavioural skills, and day. (5) The EQ was completed at the conclusion of
affirmation based learning outcomes is the next step in safety discussion. Group discussions may have had a
the development and improvement of the IS based social desirability bias effect. It is possible that some
training program. groups may have still been discussing responses on the
questionnaire with other members of the group leading
While the factor structure of the SMS 2.4 EQ is not a to bias towards opinions projected by dominate in-
measure of the SMS 2.4 program objectives or learning groups. To control for this participants could be asked
outcomes it is possible to speculate some conclusions to complete the questionnaire independently after the
about the SMS 2.4 training program. Overall there is a conclusion of training without discussion. (6) A bias
positive response towards SMS training, while a caused by fear of reprisal is likely to have increased the

6. positivity of participant responses. The request for 2. McFadden, K.L., & Towell, E.R. (1991). Aviation
employees to write their employee number on the SMS human factors: A framework for the new millennium.
2.4 EQ is likely to have inflated positive responses due Journal of Air Transport Management, 5, 177-184.
to fear of negative repercussions if reporting they did 3. Reason, J. (1995). A systems approach to
not like the training. Program instructors collecting and organizational error. Ergonomics, 38, 1708-1721.
having access to completed, identifiable questionnaires 4. Helmreich, R.L., & Foushee, H.C. (1993). Why crew
may also have contributed to this bias. Future studies resource management? In E.L. Wiener, B.G. Kanki, &
could limit this effect by not requesting employee R.L. Helmreich (Eds.), Cockpit Resource Management
numbers and allowing participants to submit EQs (pp. 3-45). San Diego, USA: Academic Press, Inc.
anonymously. 5. Helmreich, R.L., & Merritt, A.C. (1998). Culture at
work. Aldershot, UK: Avebury.
4.3 Conclusion 6. Bent, J. (2003). The under-employed full flight
simulator. Paper presented at the Sixth International
This research represents the first psychometric AAvPA Symposium in Sydney, Australia.
evaluation of the SMS 2.4 EQ. The EQ is likely a
7. Reason, J. (1997). Managing the risks of
measure of the emotive reaction towards the overall
organizational accidents. Aldershot, UK: Ashgate
training process and not the outcomes of IS as intended. Publishing Limited.
Insufficient evidence was available to suggest support as
8. Salas, E., & Cannon-Bowers, J.A. (2001). The science
to whether IS was beneficial to employee safety
of training: A decade of progress. Annual Review of
behaviour or learning. Future research is required to Psychology, 52, 471-499.
determine if there is an actual benefit to learning (and
subsequent behavioural transfer) as a result of being 9. Valverde, H.H. (1973). A review of flight simulator
transfer of training studies. HF, 15(6), 510-523.
exposed to the IS program at the ARTRC. Currently the
mixed group training structure is being negatively 10. Salas, E., Bowers, C.A., & Rhodenizer, L. (1998). It is
received by employees. Overall however, it appears the not how much you have but how you use it: Toward a
rational use of simulation to support training. IJAP,
training program is being positively received.
8(3), 197-208.
11. Bramley, P. (1991). Evaluating training effectiveness:
4.4 Industry Applications Translating theory into practice. Sydney: McGraw-
Industries dominated by simulation training are reliant Hill Book Company.
on effective training tools and structures and also the 12. Kirkpatrick, D.L. (1959). Techniques for evaluating
skilled functioning and reasoning of human operators. training programs. Journal of the American Society of
Evaluation is the process through which training Training and Development, 13, 3-9.
effectiveness is derived and the mechanism for 13. Kirkpatrick, D.L. (2004). A training and development
strategically improving training processes and classic: How to start an objective evaluation of
performance outcomes. Industrial accidents such as the your training program. Training & Development,
Glenbrook rail disaster remind us of the human cost of 58(5), 1-3.
industry and the consequence of ineffective training or 14. Goldstein, I.L. (1986). Training in organisations (2nd
training which has not been scientifically evaluated [20]. ed.). San Francisco: Brooks/Cole.
The identified lack of psychometric evaluation of 15. Arthur, J.W., Bennett, J.W., Edens, P.S., & Bell, S.T.
training within the simulation industry and general (2003). Effectiveness of training in organizations: A
industry [6],[8],[11],[14],[15],[20] does not represent a meta-analysis of design and evaluation features.
safety oriented approach to the active pursuit of Journal of Applied Psychology, 88(2), 234-245.
advancement. Strategic direction for the evaluation and 16. Schwartz, B., &. Robbins, S.J. (1995). Psychology of
subsequent improvement of training is accessible learning and behaviour. London: Norton & Co.
through best practice models such as Kirkpatrick’s 17. Machles, D. (2003). Evaluating the effectiveness of
model of training evaluation [12] and Bramley’s training safety training. OH&S, 72(6), 54.
cycle model [11]. The theoretical concepts presented in 18. Kraiger, K., Ford, J.K., & Salas, E. (1993).
these models have practical applications to industry and Application of cognitive, skill-based, and affective
direct the development of new or redevelopment of theories of learning outcomes to new methods of
existing training evaluation measures. The development training evaluation. Journal of Applied Psychology,
of valid evaluation structures is a complex process 78(2), 311-328.
reliant on psychometric examination and the revision of 19. Edkins, G.D., & Pollock, C.M. (1997). The influence
developing tools. Effective evaluation is an essential of sustained attention on railway accidents. Accident
step towards strategic management of learning, it is Analysis & Prevention, 29(4), 533-539.
essential such tools are incorporated into the wider 20. McInerney, P.A. (2001). Special Commission of
training process enabling continued and proactive Inquiry into the Glenbrook Rail Accident (Special
advancement for a progressive industry. Commission of Inquiry). Sydney, Australia.
21. Leape, L. (1994). Error in medicine. JAMA, 272(23),
REFERENCES 1851-1857.
1. Helmreich, R.L. (2000). On error management:
Lessons from aviation. BMJ, 320, 781-785.