Pspms Predictive Safety Performance Mngt Sys


Published on

The Future:Predictive Safety Performance Management System

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Pspms Predictive Safety Performance Mngt Sys

  1. 1. PSPMS an airline operational data model for a“predictive safety performance management system”P. Ulfvengren, J. RignérIndustrial Management, KTH Royal Institute of Technology, Stockholm, SwedenM.C Leva, N. McDonaldAerospace Psychology Research Group – School of Psychology Trinity College Dublin, IrelandCpt. M. YdalusHead of Safety Office STOOB Airline operations, SAS, Stockholm, SwedenIntroduction ”It takes all the running you can do, to keep in the same place. If you want to go somewhere else, you must run at least twice as fast as that.” -Lewis Carroll’s Through the Looking glassToday’s business environment is constantly changing and boundaries in the design envelope(Rasmussen, 1997) keep pushing the operational point towards the boarder of acceptable safety limit.New demands increase pressure on capacity, safety, quality, security, cost, environment, regulationsand competition. This makes the pressure in the operational process constantly changing. To keepoperating safe under these constantly new conditions requires a flexibility and control over safetyrelated performance along with other important performance measures reflecting the success of thenew demands on the system.This means that operations will have to constantly transform. In turn this increases the organizationalprocesses delivering improvement to operations get feedback of what influences the new demands on astrategic level have on the operational process.In earlier research (McDonald et al., HILAS, 2009) a theoretical framework with a human factorsapproach has been developed. It is believed that this complementary view is necessary to identify areasof improvement with the most relevance and leverage on safety critical systems change.The purpose with Management control is to have intentional impact on operations, executives andpersonnel towards certain objectives. Support for these tasks are: formal controls such as budgets and 1
  2. 2. performance measures; Organizational structure such as allocation of responsibilities andaccountabilities and incentives. Less formalized control such as organizational culture, learning andauthorization (Kulven et al., 2010).Performance management (PM) is an activity aimed at delivering improvement of performance.Traditionally performance management in business used financial measures. Using key performanceindicators at various departments allowed aggregation and collection of measures to gather an overallsummary of results of the business as well as to identify specific areas’ performance and need forimprovement. The overall goal is to improve operational process outcome.Here the performance management has a focus on performance related to a successful change and safeoperations. Performance management is one angle of management which needs to be integrated withother organizational focuses such as change and risk management.Management approaches such as Lean help a very pressed industry to survive and stay in a cost-cuttingmode. Perhaps we are moving towards a point where the system is so “stretched” and where this ismanifested in fatigue, and stress and see that production pressure impact safety? This link is not knownand possible impact on safety is manifested in a parallel management system, in reports, and SPI’s. Thisis not always known in the system measuring Lean KPIs. Safety deficiencies are still often explained by“simple” human factors such as human error and compliance to routines and human performance. 30years of HF history has not proved a concept that directs actions to real system change andimprovement. Identifying HF factors even at a systemic level (individual-technology, joint cognitivesystems and latent conditions) may help develop indicators but not identifying system antecedents thatinfluence human performance in the system operations.Industrial leaders admit they have difficulty with change but it is not the actual change that is difficult.“As soon as we know what needs to change and have an idea how it is more or less just to put it on theright track. However it is identifying what needs to change, to know it has changed and actually had thedesired effect that is difficult” (MASCA, 2011).Safety is a field that does not lend itself easily to traditional performance measures. In safety researchthere are many models explaining what happens when safety fails such as weaknesses in technology,organizations and culture. What is needed is to implement a new HF framework that may govern systemperformance that will improve safety. When logic is laid, tools and systems are required to supportsystem performance management with a system integrating risk, change and learning. (1+2)Literature review / Situation todayThe Performance Review Commission (PRC) produced a report (Eurocontrol, 2009) ATM AirportPerformance (ATMAP) Framework. IT describes a framework for performance measures for airportairside and nearby airspace performance. It is made up by the following components; Scope definitionwithin which measurements are conducted; list of factors affecting performance; 2
  3. 3. Relevant Key Performance Areas (KPAs) to describe performance objectives of an airport community;Key Performance Indicators (KPIs) to measure KPAs; Data requirements to populate KPIs and variousapproaches on how to use the KPIs in order to assess and evaluate performance. (3+4)The ARMS concept, Airline Risk Management Solutions (Nisula, 2008), has been called the nextgeneration methodology for operational risk assessment. Operational risk assessment is a core part ofthe safety management system. It is a result of a consensus among practitioners that all existingmethods had serious shortcomings. The new methodology for global risk assessment builds on fourterms rather than the traditional two in a risk matrix, namely probability and severity. Instead there arein ARMS two matrixes, which include two terms between probability and severity. The first matrixincludes frequency and avoidability and will result in a number value. The second matrix includesrecoverability and gravity and will result in a letter value. The combined assessments give values oflikelihood and severity as before in an unacceptable zone and in an acceptable zone. ARMS also presenta new framework addressing risk controls in order to prevent and avoid threats, avoid and recover fromundesired events and recover and mitigate accidents. ARMS ensure urgent issues are addressed timelyand is a practical and acceptable method that support the risk assessment phase for industry.Another recent initiative is the APF (Aerospace Performance Factor) (Eurocontrol, 2009). The workinggroup behind the APF argued that it changes how aviation views on safety information and expectsimprovements in competitive efficiencies, improvements in insurance costs and a step towardsoperational risk forecasting as a result. The idea is to integrate safety information into organizationaland operational decision making by breaking down silos and then merge data sources. This allows safetyinformation to come to full field of view which is argued to be proactive and enhance risk forecasting.The main process is around organizing and prioritizing safety information. Safety data and informationsources are mapped with regard to origin in operations and system level. Then a pair wise comparisonsat all levels is done by use of elements of Analytical Hierarchy Processing (AHP). This methodology isused in multi-objective decision making analysis. The last step in the process is to display information fordecision makers: Various graphs are produced that may visualize various levels leading up to the APFlevel. The APF is the sum of weighted occurrences over appropriate denominator. The denominator ischosen appropriate to the domain, i.e. Eurocontrol use the number of flight hours flown in the timeinterval of interest, but for an airline it might be the number of flight sectors flown. (5)ICAO has produced two main draft documents of a Safety Management Manual (SMM) in 2006 and2008 (ICAO, 2006, 2008). The manual is intended to provide States with guidance to develop aregulatory framework and guidance material for implementation of a Safety Management System (SMS)by service providers as well as for the implementation of a State Safety Program (SSP). Only SMS will bediscussed here.Guidance is given for SMS planning and operations following ICAO SMS framework:  Safety policy and objective,  Safety risk management,  Safety assurance and  Safety promotion. 3
  4. 4. States shall require an SMS from service providers that are exposed to safety risks. In the Annexes to theICAO SMS document it is also established that the SMS shall be accepted by the state. The SMS shall, asa minimum:  identify safety hazards;  ensure the implementation of remedial action necessary to maintain agreed safety performance;  provide for continuous monitoring and regular assessment of the safety performance; and  aim at a continuous improvement of the overall performance of the safety management system. The notion of Safety performance is expressed as an essential ingredient of the effectiveoperation of an SMS and progress towards a performance-based regulatory environment. Safetyperformance for service providers are proposed to be expressed in terms of compliance to agreed onand set safety requirements, safety performance targets and safety performance indicators. “The safety performance of an SMS is not directly related to the quantification of high- consequence outcomes (safety measurement) but rather to the quantification of the low- consequence process (safety performance measurement)”. Safety performance indicators are described as short-term, tactical, measurable objectivesreflecting the safety performance of an SMS. They are expressed in numerical terms; they should beobvious, measurable and linked to the safety concerns of an SMS. Safety performance indicators reflectsafety performance measurement exclusively. The safety performance indicators of an SMS should notreflect safety measurement. State and individual organizations will separately agree on safetyperformance targets for each aviation provider, therefore safety performance indicators may differbetween individual service providers. Safety performance targets are described, by ICAO, as long-term, strategic measurableobjectives reflecting the safety performance of an SMS. Safety performance targets are expressed innumerical terms, they should be obvious, measurable, acceptable to stakeholders, and be linked to thesafety performance indicator (short-term objective) of an SMS. Safety requirements are described as the tools and means needed to achieve the safetyperformance indicators and safety performance targets of an SMS. This may further be interpreted aswhat is required to do or to implement to achieve the level of safety performance of the process,relating to the safety performance indicator. Indicators and targets may be the same or may be different. ICAO uses a few examplesexplaining that safety performance indicators are set in agreement between the individual serviceprovider and the state. Resources, costs and urgency for a safety performance indicator to improvetowards a more demanding target may vary and sometimes it may be considered enough to maintainthe safety performance. In this case the target and indicator are the same. In one example a low consequence process is described as number of foreign objects debris(FOD) over a period of time or for a number of flights. A range of different safety parameter indicators and targets are believed to provide a betterinsight. The SPI must be agreed upon with the authority to be reasonable and relevant for strategicmanagement safety training, believed to contribute to increased safety by improving what is identifiedas prioritized processes and assumed to reduce latent conditions. 4
  5. 5. Safety performance for service providers are proposed to be expressed in terms of compliance to agreedon and set safety requirements, safety performance targets and safety performance indicators. As perICAO(2008, p. 204) safety performance monitoring and measurement shall include the following:  hazard reporting; (mandatory, voluntary and confidential reporting)  safety studies;  safety reviews; (conducted during introduction and deployment of new technologies, change or implementation of procedures etc.)  audits;  safety surveys; and  internal safety investigations.Reactive, proactive and predictive safety management perspectives are discussed in the second editionof the guidelines and framework from ICAO (2008). The traditional approach has been to react toundesirable events by investigations finding cause and then through regulations and rules avoid ithappens again. Today, ICAO (2008) makes a distinction where reactive method responds to the eventsthat already happened, such as incidents and accidents.The proactive method “looks actively for the identification of safety risks through the analysis of theorganisations activities”. The predictive method captures system performance as it happens in real-timenormal operations to identify potential future problems.The ICAO Safety Management Manual (SMM) draft documents have been on the table and discussed bymost service providers and authorities for some time and they are supposed to influence regulationsworldwide. The ICAO manual will be the basis for EASA when formulating common requirements inEurope for SMSs and for national civil aviation authorities in their function as national regulators. Twoaspects of the proposed ICAO safety management system framework distinguish it from previousorganisational frameworks for managing safety. These concern, first, the integration of the managementof safety into the overall management system of the organisaton; and second that this should be aperformance-based regulation, capable of demonstrating its implementation and effectiveness in termsof measurable operational outcomes which are related to safety. The previous generation of safetyregulation was based on the requirement for an independent quality or safety system with separateaccountability to the accountable manager for the operational organisation. While this sought to ensurethat commercial interests do not override those of safety, the critical weaknesses of this form ofregulation concern the implementation of safety recommendations to achieve verifiable change, andthe difficulty of ensuring that a safety management system, which might look fine on paper, is actuallyworking in practice.The new generation of SMSThe new generation of SMS seeks to implement a performance-based regulation in which it isrecognised that modern organisations require greater integration of their management processes (, operations, standardisation and control functions) to deliver better outcomes overall,simultaneously satisfying commercial, quality and safety requirements. Within this framework the safetyfunctions have to deliver accountability for safety outcomes, as well as demonstrating the integrity ofsafety management processes. This creates a paradox for regulation: it has to prescribe what has to be 5
  6. 6. done to assure that safety is properly addressed; but actually assuring this requires a broader focus thanjust on safety-related functions, because safety performance depends on the integrated performance ofthe whole system; however regulators cannot prescribe what is beyond their remit which relates tothose aspects which are (specifically) related to the safety of the system. Therefore, in order tounderstand how to implement an effective safety management system we need a model that goes along way beyond the borders of what has traditionally been included under the rubric of safetymanagement.The HILAS project, while one of its drivers has always been the improvement of safety (most particularlywith respect to the human factor), has taken just such a broad remit. It has developed a platform for theintegration of human requirements across the industry lifecycle – which includes the organization,management and improvement of flight operations and aircraft maintenance (not just the safetyaspects). The HILAS management system has taken an integrated approach to the management ofperformance, risk and organizational learning and change, thus it addresses precisely the basis for theperformance outcomes required by the ICAO framework, but does so in a way that is not specific tosafety, but relates to the effective functioning of the organizational processes in delivering all theiroperational goals.HILAS has also provided the opportunity to examine in-depth other fundamental challenges ofregulation in complex systems, including: How to assess risk in ‘ultra-safe’ systems How to develop integrated risk management across multi-organizational ‘system-of-systems’ like aviation How to bridge the gap between knowing what needs to be done to improve the system and actually doing it. How to address the cultural aspects of implementing organizational and management systems across diverse regions of the world.A core challenge to developing a performance-based regulatory framework in aviation is the fact thataviation is what is often termed an ‘ultra-safe’ system in that it is impossible to differentiate betweenorganisations by means of safety failures – they are, fortunately, too few. How do we then differentiatebetween relatively safe and unsafe performance? What can be monitored is the ‘normal variation’ ofoperational performance (reports, audits, flight data, etc), on the basis of which we draw inferencesabout the probability of system failure. This assumes a linear relationship between minor failure andmajor catastrophe. Without an understanding of the underlying system mechanisms, it is doubtfulwhether this is sufficient to anticipate or prevent complex system failures. What is needed is a morecomplex in-depth analysis of system performance in order to understand both what has happened inthe past and how to anticipate future potential failure – often in the context of changed circumstances.This analysis needs to draw in the different organisations who make up the operational system (flightoperations, maintenance, air traffic, airports, etc.). If we do not achieve this, the danger is that we willbe monitoring Safety Performance Indicators which do not adequately reflect the state of the system. 6
  7. 7. Because transparent and accountable implementation and change have always been problematic issuesin safety management and its regulation, HILAS has placed a great deal of emphasis on the necessarysequence leading to system change, as follows: Providing the tools to allow all stakeholders to contribute to adequate solutions to complex system problems  Assurance that appropriate change has actually happened following recommendations  Establishing that the outcome has improved (i.e. that risk has been measurably reduced)  Enabling other organisations to learn effectively so that the system as a whole improves.These stages define the type of performance outcomes which need to be regulated, if regulation is to beeffective. (6+7+8+9)The research gapAviation is known to have enormous amount of data and to develop indicators are not the challenge.The ATMAP has a well founded performance measurement framework but aims mainly to assess andevaluate performance. In both ARMS and APF focus lies on identifying risks, acquiring data, andassessing risk and at the most gets the management attention and decision support to act timely.The real challenge is to manage the system outcome by letting the indicators drive change. What to doafter risks are identified, classified, prioritized and performance is evaluated and not sufficient? What todo when action is needed? Clearly it is ICAOs intention to move away from assessing safety merely onnumbers on operational outcome. This proposal is arguing for the research gap on the essential andundisputable need for a proven change capability in order to discuss safety performance measures. Ameasure on safety performance could be “time from identifying risk to measurable change inoperational data”. The result of an effective improvement will be identified and measured and visual inthe ARMS, APF and ATMAP but none of the concepts support any industry to achieve improvements.Since SMS assess safety performance it would be logical to develop a safety performance managementsystem. However safety and safety performance should not be managed in isolation. Performancemanagement for continuous innovation in aviation is a concept better describing the integration ofchange and in the aviation domain that by nature is a high risk industry with safety risks.Concern for the future safety of the aviation system requires us to address many issues in which triedand tested processes; methods and tools are not well known and generally available. These issuesinclude, for example: How to monitor the performance of complex systems in a valid manner How to ensure effective learning and change both within and between organisations How to ensure ‘systems-of-systems’ like aviation achieve goals in an integrated way, and how to design future integrated aviation systems to deliver operational safety requirements. 7
  8. 8. How to regulate for a global system which involves many different types of organisations, in different regions, with different cultures, interacting with each other.Performance based regulationsDeveloping performance-based regulation for safety management inevitably will touch on these issues ifsuch regulation is to play an effective role in ensuring and improving aviation safety. The HILASframework provides ways of addressing these issues that not only fulfills the Safety ManagementSystem requirements of ICAO, but also demonstrates the integration of SMS in an overall operationalmanagement concept and shows how this can foster learning between organizations and innovation forsafer systems in the future.Another identified gap is that existing management systems and financial governance and approachessuch as lean are all made for managing profitability, competitiveness and survival, but this is not enoughfor safety of or managing safety performance.The PSPMS (predictive safety performance management system) needs to fill a gap of other theoreticalframeworks that may be used to build a management structure, which address safety risks andoperational performance in an integrated manner. A Key Performance Area for the Aviation TransportSystem for the future is innovation in human systems and change in high risk industries. 8
  9. 9. Performance Management cycle for PSPMS: 1. A performance measure and monitoring framework. 2. Capability to aggregate data to appropriate level for decision makers. 3. A structural and procedural framework that uses this data for strategic decision making and priority. 4. A structural and procedural framework to de-aggregate the data and triggering appropriate change process and team in relation to the triggering performance measure(s). 5. De-aggregation should triggering appropriate change process and team in relation to the triggering performance measure(s). 6. Complex system analysis including process modeling and analysis, evaluating and developing SPI’s to identify antecedents and suggest mitigating action. 7. Implement change, using CMS with Knowledge Transformation Process, appropriate learning and training and integration with other ongoing change initiatives. 8. Evaluation framework. Develop new SPIs to monitor progress and measure Safety performance. 1. A structural and procedural framework that uses this data for strategic decision making and priority. 2. A structural and procedural framework to de-3. Capability to aggregate data to appropriate level aggregate the data for decision makers. 5. De-aggregation should triggering appropriate change 4. A performance measure and process and team in relation to the triggering performance monitoring framework. measure(s). 8. Evaluation framework. Develop new SPIs to monitor progress and 6. Complex system analysis including process modeling and analysis, measure Safety performance. evaluating and developing SPI’s to identify antecedents and suggest mitigating action. 7. Implement change, using CMS with Knowledge Transformation Process, appropriate learning and training and integration with other ongoing change initiatives. 9
  10. 10. References: 1. European Aviation Safety Plan 2011-2014 (EASA, 2011). Annual Safety review 2010, (EASA, 2010). 2. Flight path 2050. 3. Human Factors performance indicators for the energy and related process industries (Energy Institute, 2010). 4. Developing process safety indicators (HSE, 2009). 5. Resilience engineering (Eurocontrol, 2009). 6. Understanding and Evaluating the Federal Aviation Administration Safety Oversight system (Hansen et al., 2006), 7. History of Aviation Safety Oversight in the US (Hansen et al., 2005). 8. Human Performance in Air Traffic Management Safety (Eurocontrol, 2010). 9. New HF (McDonald et al. 2009) 10