The Future:Predictive Safety Performance Management System

1. PSPMS an airline operational data model for a
“predictive safety performance management system”
P. Ulfvengren, J. Rignér
Industrial Management, KTH Royal Institute of Technology, Stockholm, Sweden
M.C Leva, N. McDonald
Aerospace Psychology Research Group – School of Psychology Trinity College Dublin, Ireland
Cpt. M. Ydalus
Head of Safety Office STOOB Airline operations, SAS, Stockholm, Sweden
Introduction
”It takes all the running you can do, to keep in the same place. If you want to go
somewhere else, you must run at least twice as fast as that.”
-Lewis Carroll’s Through the Looking glass
Today’s business environment is constantly changing and boundaries in the design envelope
(Rasmussen, 1997) keep pushing the operational point towards the boarder of acceptable safety limit.
New demands increase pressure on capacity, safety, quality, security, cost, environment, regulations
and competition. This makes the pressure in the operational process constantly changing. To keep
operating safe under these constantly new conditions requires a flexibility and control over safety
related performance along with other important performance measures reflecting the success of the
new demands on the system.
This means that operations will have to constantly transform. In turn this increases the organizational
processes delivering improvement to operations get feedback of what influences the new demands on a
strategic level have on the operational process.
In earlier research (McDonald et al., HILAS, 2009) a theoretical framework with a human factors
approach has been developed. It is believed that this complementary view is necessary to identify areas
of improvement with the most relevance and leverage on safety critical systems change.
The purpose with Management control is to have intentional impact on operations, executives and
personnel towards certain objectives. Support for these tasks are: formal controls such as budgets and
1

2. performance measures; Organizational structure such as allocation of responsibilities and
accountabilities and incentives. Less formalized control such as organizational culture, learning and
authorization (Kulven et al., 2010).
Performance management (PM) is an activity aimed at delivering improvement of performance.
Traditionally performance management in business used financial measures. Using key performance
indicators at various departments allowed aggregation and collection of measures to gather an overall
summary of results of the business as well as to identify specific areas’ performance and need for
improvement. The overall goal is to improve operational process outcome.
Here the performance management has a focus on performance related to a successful change and safe
operations. Performance management is one angle of management which needs to be integrated with
other organizational focuses such as change and risk management.
Management approaches such as Lean help a very pressed industry to survive and stay in a cost-cutting
mode. Perhaps we are moving towards a point where the system is so “stretched” and where this is
manifested in fatigue, and stress and see that production pressure impact safety? This link is not known
and possible impact on safety is manifested in a parallel management system, in reports, and SPI’s. This
is not always known in the system measuring Lean KPIs. Safety deficiencies are still often explained by
“simple” human factors such as human error and compliance to routines and human performance. 30
years of HF history has not proved a concept that directs actions to real system change and
improvement. Identifying HF factors even at a systemic level (individual-technology, joint cognitive
systems and latent conditions) may help develop indicators but not identifying system antecedents that
influence human performance in the system operations.
Industrial leaders admit they have difficulty with change but it is not the actual change that is difficult.
“As soon as we know what needs to change and have an idea how it is more or less just to put it on the
right track. However it is identifying what needs to change, to know it has changed and actually had the
desired effect that is difficult” (MASCA, 2011).
Safety is a field that does not lend itself easily to traditional performance measures. In safety research
there are many models explaining what happens when safety fails such as weaknesses in technology,
organizations and culture. What is needed is to implement a new HF framework that may govern system
performance that will improve safety. When logic is laid, tools and systems are required to support
system performance management with a system integrating risk, change and learning. (1+2)
Literature review / Situation today
The Performance Review Commission (PRC) produced a report (Eurocontrol, 2009) ATM Airport
Performance (ATMAP) Framework. IT describes a framework for performance measures for airport
airside and nearby airspace performance. It is made up by the following components; Scope definition
within which measurements are conducted; list of factors affecting performance;
2

3. Relevant Key Performance Areas (KPAs) to describe performance objectives of an airport community;
Key Performance Indicators (KPIs) to measure KPAs; Data requirements to populate KPIs and various
approaches on how to use the KPIs in order to assess and evaluate performance. (3+4)
The ARMS concept, Airline Risk Management Solutions (Nisula, 2008), has been called the next
generation methodology for operational risk assessment. Operational risk assessment is a core part of
the safety management system. It is a result of a consensus among practitioners that all existing
methods had serious shortcomings. The new methodology for global risk assessment builds on four
terms rather than the traditional two in a risk matrix, namely probability and severity. Instead there are
in ARMS two matrixes, which include two terms between probability and severity. The first matrix
includes frequency and avoidability and will result in a number value. The second matrix includes
recoverability and gravity and will result in a letter value. The combined assessments give values of
likelihood and severity as before in an unacceptable zone and in an acceptable zone. ARMS also present
a new framework addressing risk controls in order to prevent and avoid threats, avoid and recover from
undesired events and recover and mitigate accidents. ARMS ensure urgent issues are addressed timely
and is a practical and acceptable method that support the risk assessment phase for industry.
Another recent initiative is the APF (Aerospace Performance Factor) (Eurocontrol, 2009). The working
group behind the APF argued that it changes how aviation views on safety information and expects
improvements in competitive efficiencies, improvements in insurance costs and a step towards
operational risk forecasting as a result. The idea is to integrate safety information into organizational
and operational decision making by breaking down silos and then merge data sources. This allows safety
information to come to full field of view which is argued to be proactive and enhance risk forecasting.
The main process is around organizing and prioritizing safety information. Safety data and information
sources are mapped with regard to origin in operations and system level. Then a pair wise comparisons
at all levels is done by use of elements of Analytical Hierarchy Processing (AHP). This methodology is
used in multi-objective decision making analysis. The last step in the process is to display information for
decision makers: Various graphs are produced that may visualize various levels leading up to the APF
level. The APF is the sum of weighted occurrences over appropriate denominator. The denominator is
chosen appropriate to the domain, i.e. Eurocontrol use the number of flight hours flown in the time
interval of interest, but for an airline it might be the number of flight sectors flown. (5)
ICAO has produced two main draft documents of a Safety Management Manual (SMM) in 2006 and
2008 (ICAO, 2006, 2008). The manual is intended to provide States with guidance to develop a
regulatory framework and guidance material for implementation of a Safety Management System (SMS)
by service providers as well as for the implementation of a State Safety Program (SSP). Only SMS will be
discussed here.
Guidance is given for SMS planning and operations following ICAO SMS framework:
 Safety policy and objective,
 Safety risk management,
 Safety assurance and
 Safety promotion.
3

4. States shall require an SMS from service providers that are exposed to safety risks. In the Annexes to the
ICAO SMS document it is also established that the SMS shall be accepted by the state. The SMS shall, as
a minimum:
 identify safety hazards;
 ensure the implementation of remedial action necessary to maintain agreed safety
performance;
 provide for continuous monitoring and regular assessment of the safety performance;
and
 aim at a continuous improvement of the overall performance of the safety management
system.
The notion of Safety performance is expressed as an essential ingredient of the effective
operation of an SMS and progress towards a performance-based regulatory environment. Safety
performance for service providers are proposed to be expressed in terms of compliance to agreed on
and set safety requirements, safety performance targets and safety performance indicators.
“The safety performance of an SMS is not directly related to the quantification of high-
consequence outcomes (safety measurement) but rather to the quantification of the low-
consequence process (safety performance measurement)”.
Safety performance indicators are described as short-term, tactical, measurable objectives
reflecting the safety performance of an SMS. They are expressed in numerical terms; they should be
obvious, measurable and linked to the safety concerns of an SMS. Safety performance indicators reflect
safety performance measurement exclusively. The safety performance indicators of an SMS should not
reflect safety measurement. State and individual organizations will separately agree on safety
performance targets for each aviation provider, therefore safety performance indicators may differ
between individual service providers.
Safety performance targets are described, by ICAO, as long-term, strategic measurable
objectives reflecting the safety performance of an SMS. Safety performance targets are expressed in
numerical terms, they should be obvious, measurable, acceptable to stakeholders, and be linked to the
safety performance indicator (short-term objective) of an SMS.
Safety requirements are described as the tools and means needed to achieve the safety
performance indicators and safety performance targets of an SMS. This may further be interpreted as
what is required to do or to implement to achieve the level of safety performance of the process,
relating to the safety performance indicator.
Indicators and targets may be the same or may be different. ICAO uses a few examples
explaining that safety performance indicators are set in agreement between the individual service
provider and the state. Resources, costs and urgency for a safety performance indicator to improve
towards a more demanding target may vary and sometimes it may be considered enough to maintain
the safety performance. In this case the target and indicator are the same.
In one example a low consequence process is described as number of foreign objects debris
(FOD) over a period of time or for a number of flights.
A range of different safety parameter indicators and targets are believed to provide a better
insight. The SPI must be agreed upon with the authority to be reasonable and relevant for strategic
management safety training, believed to contribute to increased safety by improving what is identified
as prioritized processes and assumed to reduce latent conditions.
4

5. Safety performance for service providers are proposed to be expressed in terms of compliance to agreed
on and set safety requirements, safety performance targets and safety performance indicators. As per
ICAO(2008, p. 204) safety performance monitoring and measurement shall include the following:
 hazard reporting; (mandatory, voluntary and confidential reporting)
 safety studies;
 safety reviews; (conducted during introduction and deployment of new technologies, change or
implementation of procedures etc.)
 audits;
 safety surveys; and
 internal safety investigations.
Reactive, proactive and predictive safety management perspectives are discussed in the second edition
of the guidelines and framework from ICAO (2008). The traditional approach has been to react to
undesirable events by investigations finding cause and then through regulations and rules avoid it
happens again. Today, ICAO (2008) makes a distinction where reactive method responds to the events
that already happened, such as incidents and accidents.
The proactive method “looks actively for the identification of safety risks through the analysis of the
organisations activities”. The predictive method captures system performance as it happens in real-time
normal operations to identify potential future problems.
The ICAO Safety Management Manual (SMM) draft documents have been on the table and discussed by
most service providers and authorities for some time and they are supposed to influence regulations
worldwide. The ICAO manual will be the basis for EASA when formulating common requirements in
Europe for SMSs and for national civil aviation authorities in their function as national regulators. Two
aspects of the proposed ICAO safety management system framework distinguish it from previous
organisational frameworks for managing safety. These concern, first, the integration of the management
of safety into the overall management system of the organisaton; and second that this should be a
performance-based regulation, capable of demonstrating its implementation and effectiveness in terms
of measurable operational outcomes which are related to safety. The previous generation of safety
regulation was based on the requirement for an independent quality or safety system with separate
accountability to the accountable manager for the operational organisation. While this sought to ensure
that commercial interests do not override those of safety, the critical weaknesses of this form of
regulation concern the implementation of safety recommendations to achieve verifiable change, and
the difficulty of ensuring that a safety management system, which might look fine on paper, is actually
working in practice.
The new generation of SMS
The new generation of SMS seeks to implement a performance-based regulation in which it is
recognised that modern organisations require greater integration of their management processes (e.g.
supply, operations, standardisation and control functions) to deliver better outcomes overall,
simultaneously satisfying commercial, quality and safety requirements. Within this framework the safety
functions have to deliver accountability for safety outcomes, as well as demonstrating the integrity of
safety management processes. This creates a paradox for regulation: it has to prescribe what has to be
5

6. done to assure that safety is properly addressed; but actually assuring this requires a broader focus than
just on safety-related functions, because safety performance depends on the integrated performance of
the whole system; however regulators cannot prescribe what is beyond their remit which relates to
those aspects which are (specifically) related to the safety of the system. Therefore, in order to
understand how to implement an effective safety management system we need a model that goes a
long way beyond the borders of what has traditionally been included under the rubric of safety
management.
The HILAS project, while one of its drivers has always been the improvement of safety (most particularly
with respect to the human factor), has taken just such a broad remit. It has developed a platform for the
integration of human requirements across the industry lifecycle – which includes the organization,
management and improvement of flight operations and aircraft maintenance (not just the safety
aspects). The HILAS management system has taken an integrated approach to the management of
performance, risk and organizational learning and change, thus it addresses precisely the basis for the
performance outcomes required by the ICAO framework, but does so in a way that is not specific to
safety, but relates to the effective functioning of the organizational processes in delivering all their
operational goals.
HILAS has also provided the opportunity to examine in-depth other fundamental challenges of
regulation in complex systems, including:
How to assess risk in ‘ultra-safe’ systems
How to develop integrated risk management across multi-organizational ‘system-of-systems’
like aviation
How to bridge the gap between knowing what needs to be done to improve the system and
actually doing it.
How to address the cultural aspects of implementing organizational and management systems
across diverse regions of the world.
A core challenge to developing a performance-based regulatory framework in aviation is the fact that
aviation is what is often termed an ‘ultra-safe’ system in that it is impossible to differentiate between
organisations by means of safety failures – they are, fortunately, too few. How do we then differentiate
between relatively safe and unsafe performance? What can be monitored is the ‘normal variation’ of
operational performance (reports, audits, flight data, etc), on the basis of which we draw inferences
about the probability of system failure. This assumes a linear relationship between minor failure and
major catastrophe. Without an understanding of the underlying system mechanisms, it is doubtful
whether this is sufficient to anticipate or prevent complex system failures. What is needed is a more
complex in-depth analysis of system performance in order to understand both what has happened in
the past and how to anticipate future potential failure – often in the context of changed circumstances.
This analysis needs to draw in the different organisations who make up the operational system (flight
operations, maintenance, air traffic, airports, etc.). If we do not achieve this, the danger is that we will
be monitoring Safety Performance Indicators which do not adequately reflect the state of the system.
6

7. Because transparent and accountable implementation and change have always been problematic issues
in safety management and its regulation, HILAS has placed a great deal of emphasis on the necessary
sequence leading to system change, as follows:
Providing the tools to allow all stakeholders to contribute to adequate solutions to
complex system problems
 Assurance that appropriate change has actually happened following recommendations
 Establishing that the outcome has improved (i.e. that risk has been measurably reduced)
 Enabling other organisations to learn effectively so that the system as a whole improves.
These stages define the type of performance outcomes which need to be regulated, if regulation is to be
effective. (6+7+8+9)
The research gap
Aviation is known to have enormous amount of data and to develop indicators are not the challenge.
The ATMAP has a well founded performance measurement framework but aims mainly to assess and
evaluate performance. In both ARMS and APF focus lies on identifying risks, acquiring data, and
assessing risk and at the most gets the management attention and decision support to act timely.
The real challenge is to manage the system outcome by letting the indicators drive change. What to do
after risks are identified, classified, prioritized and performance is evaluated and not sufficient? What to
do when action is needed? Clearly it is ICAOs intention to move away from assessing safety merely on
numbers on operational outcome. This proposal is arguing for the research gap on the essential and
undisputable need for a proven change capability in order to discuss safety performance measures. A
measure on safety performance could be “time from identifying risk to measurable change in
operational data”. The result of an effective improvement will be identified and measured and visual in
the ARMS, APF and ATMAP but none of the concepts support any industry to achieve improvements.
Since SMS assess safety performance it would be logical to develop a safety performance management
system. However safety and safety performance should not be managed in isolation. Performance
management for continuous innovation in aviation is a concept better describing the integration of
change and in the aviation domain that by nature is a high risk industry with safety risks.
Concern for the future safety of the aviation system requires us to address many issues in which tried
and tested processes; methods and tools are not well known and generally available. These issues
include, for example:
How to monitor the performance of complex systems in a valid manner
How to ensure effective learning and change both within and between organisations
How to ensure ‘systems-of-systems’ like aviation achieve goals in an integrated way, and
how to design future integrated aviation systems to deliver operational safety
requirements.
7

8. How to regulate for a global system which involves many different types of
organisations, in different regions, with different cultures, interacting with each other.
Performance based regulations
Developing performance-based regulation for safety management inevitably will touch on these issues if
such regulation is to play an effective role in ensuring and improving aviation safety. The HILAS
framework provides ways of addressing these issues that not only fulfills the Safety Management
System requirements of ICAO, but also demonstrates the integration of SMS in an overall operational
management concept and shows how this can foster learning between organizations and innovation for
safer systems in the future.
Another identified gap is that existing management systems and financial governance and approaches
such as lean are all made for managing profitability, competitiveness and survival, but this is not enough
for safety of or managing safety performance.
The PSPMS (predictive safety performance management system) needs to fill a gap of other theoretical
frameworks that may be used to build a management structure, which address safety risks and
operational performance in an integrated manner. A Key Performance Area for the Aviation Transport
System for the future is innovation in human systems and change in high risk industries.
8

9. Performance Management cycle for PSPMS:
1. A performance measure and monitoring framework.
2. Capability to aggregate data to appropriate level for decision makers.
3. A structural and procedural framework that uses this data for strategic decision making and
priority.
4. A structural and procedural framework to de-aggregate the data and triggering appropriate
change process and team in relation to the triggering performance measure(s).
5. De-aggregation should triggering appropriate change process and team in relation to the
triggering performance measure(s).
6. Complex system analysis including process modeling and analysis, evaluating and developing
SPI’s to identify antecedents and suggest mitigating action.
7. Implement change, using CMS with Knowledge Transformation Process, appropriate learning
and training and integration with other ongoing change initiatives.
8. Evaluation framework. Develop new SPIs to monitor progress and measure Safety performance.
1. A structural and procedural framework that uses this data
for strategic decision making and priority.
2. A structural and procedural framework to de-
3. Capability to aggregate data to appropriate level aggregate the data
for decision makers.
5. De-aggregation should triggering appropriate change
4. A performance measure and
process and team in relation to the triggering performance
monitoring framework.
measure(s).
8. Evaluation framework. Develop
new SPIs to monitor progress and 6. Complex system analysis including process modeling and analysis,
measure Safety performance. evaluating and developing SPI’s to identify antecedents and suggest
mitigating action.
7. Implement change, using CMS with Knowledge Transformation Process,
appropriate learning and training and integration with other ongoing change
initiatives.
9

10. References:
1. European Aviation Safety Plan 2011-2014 (EASA, 2011). Annual Safety review 2010,
(EASA, 2010).
2. Flight path 2050.
3. Human Factors performance indicators for the energy and related process industries
(Energy Institute, 2010).
4. Developing process safety indicators (HSE, 2009).
5. Resilience engineering (Eurocontrol, 2009).
6. Understanding and Evaluating the Federal Aviation Administration Safety Oversight
system (Hansen et al., 2006),
7. History of Aviation Safety Oversight in the US (Hansen et al., 2005).
8. Human Performance in Air Traffic Management Safety (Eurocontrol, 2010).
9. New HF (McDonald et al. 2009)
10