Safety Management Systems (SMS) Fundamentals: Safety Assurance
•Download as PPT, PDF•
13 likes•6,026 views
The document discusses safety assurance, which involves collecting information to ensure that safety risk controls are effective and continuously meeting objectives. It describes safety assurance functions like monitoring operations, conducting audits, analyzing data from sources like reports and investigations, assessing system performance, and implementing corrective actions. The goal is to give confidence that risk controls are working as intended and safety is being maintained. Continuous monitoring, audits, data analysis, and corrective actions form a process circle to ensure safety.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (18)
Similar to Safety Management Systems (SMS) Fundamentals: Safety Assurance
Similar to Safety Management Systems (SMS) Fundamentals: Safety Assurance (20)
More from FAA Safety Team Central Florida
More from FAA Safety Team Central Florida (20)
Recently uploaded
Recently uploaded (20)
Safety Management Systems (SMS) Fundamentals: Safety Assurance
- 1. SMS Details: Federal Aviation Administration Safety Assurance Component Policy (Structure) Risk Management Safety Assurance Safety Promotion Federal Aviation Administration SL-1
- 2. Objective of Safety Assurance • Assurance: “something that gives confidence”1 • Safety Assurance: “something that gives confidence in safety risk controls.” 1 Black’s Law Dictionary Federal Aviation Administration SL-2
- 3. Safety Assurance Functions: • Collect and analyze information to determine that process requirements are continuously being met. • Assess performance and effectiveness of risk controls. • Works in partnership with Safety Risk Management (SRM). AC 120-92 Federal Aviation Administration SL-3
- 4. S.A. is similar to Q.A. • QA focuses on product conformity & customer satisfaction on a continuous basis. • SA ensures that risk controls, once designed and put to place, perform in a way that continue to meet their safety objectives. • Integration of management systems may be beneficial. AC 120-92 Federal Aviation Administration SL-4
- 5. System Monitoring System Operation Data Acquisition & Process Written documentation to describe: Analysis System Assmt Who, What, When, Where, Why, How Preventive/ Corrective Action The system monitoring includes: 1) System description, including risk controls added during SRM. This forms the basis for SA functions such as audits and analysis. 2) Monitoring of risk controls during operations, AC 120-92 Federal Aviation Administration SL-5
- 6. Data Acquisition and Processing System Operation Data Acquisition & Process Types of Data Sources 1. 2. 3. 4. 5. 6. Continuous Monitoring Internal Audits Internal Evaluation External Audits Investigations Employee Reporting Systems Analysis System Assmt Preventive/ Corrective Action SMS Framework Elements 3.1.1 through 3.1.6 Federal Aviation Administration SL-6
- 7. Continuous Monitoring System Operation Where SRM and SA interface - risk controls Data Acquisition & Process Analysis Line managers of operational departments: • Accomplish continuous monitoring of day-to-day activities & processes System Assmt Preventive/ Corrective Action • Have direct responsibility for process control • Must ensure that processes in their areas function as designed. Federal Aviation Administration SL-7
- 8. Continuous Monitoring - Operational Data Sources • Flight dispatch records • Flight schedules • Financial data • Crew schedules and records • Warranty return reports • Aircraft discrepancy reports • Flight cancellation and delay reports Federal Aviation Administration SL-8
- 9. Internal Audits System Operation The day-to-day responsibility for safety management rests with those who “own” the technical processes. Data Acquisition & Process Analysis System Assmt Preventive/ Corrective Action This is where: • Deficiencies in processes contribute to risk • Audits provide feedback to process owners • Direct supervisory control and resource allocation can help to maintain effectiveness of risk controls Federal Aviation Administration SL-9
- 10. Internal Audit Design System Operation Data Acquisition & Process • Performed by each department. • Department Director/Manager is responsible. Analysis System Assmt • Regularly scheduled Preventive/ Corrective Action • Include contractors & vendors • Determine: – Conformity with safety risk controls – Performance of safety risk controls – Performance to meet safety objectives • Deficiencies always get action! AC 120-92 Federal Aviation Administration SL-10
- 11. Internal Evaluation System Operation • Performed by a functionally independent person or organization (e.g. QA, Safety) Data Acquisition & Process Analysis System Assmt • A process-oriented control function Preventive/ Corrective Action • Backs up the internal audit function • Uses sampling to validate SA processes Federal Aviation Administration SL-11
- 12. External Audits System Operation Data Acquisition & Process Analysis Conducted by: System Assmt • Code-share partners • Industry organizations (e.g. C.A.S.E.) Preventive/ Corrective Action • Third parties: consultants • The regulator (FAA) = “Safety Oversight” Federal Aviation Administration SL-12
- 13. Safety investigations System Operation Data Acquisition & Process • Traditional Approach – put the event behind us (everything is okay): – To reassert trust and faith in the system (as is) – To resume normal activities – To fulfil political purposes Analysis System Assmt Preventive/ Corrective Action • SMS Approach - Improved system reliability: – To learn about system vulnerability – To develop strategies for change – To prioritize investment of resources Federal Aviation Administration SL-13
- 14. Employee Reporting System Operation • Employee safety reporting is required & a feedback system is encouraged. Data Acquisition & Process Analysis System Assmt • Must provide confidentiality. • Employees must be encouraged to use the system. Preventive/ Corrective Action • Data may identify emerging hazards. • Data must be included in analysis. AC 120-92, App. 1 Federal Aviation Administration SL-14
- 15. Types of analysis System Operation Data Acquisition & Process • Against criteria/objectives Analysis • Compared to “norms” System Assmt • Patterns from multiple data points Preventive/ Corrective Action • Trends over time – “Trends” is one of the most misused term in analysis – Must have stable, reliable measures at each time sample for a valid trend Federal Aviation Administration SL-16
- 16. System Assessment System Operation • Are objectives being met? (“Happy loop”) • Risk controls failing due to: – – – – Lack of supervision Lack of resources Lack of training Poor job aids Data Acquisition & Process Analysis System Assmt Preventive/ Corrective Action • New Hazard/failed Risk Controls (Redesign - back to SRM) • Prioritize according to safety criticality (triage) Federal Aviation Administration SL-17
- 17. Preventive/Corrective Actions System Operation • Increased supervision Data Acquisition & Process • Emphasized procedures Analysis • Equipment maintenance System Assmt • Remedial training Preventive/ Corrective Action • Schedule changes • Personnel assignments • Health and wellness Federal Aviation Administration SL-18
- 18. Management Review Top management must conduct regular reviews of the SMS, including: • The outputs of SRM & SA • Lessons learned • Need for changes Federal Aviation Administration SL-19
- 19. Continuous Improvement The organization shall continuously improve the effectiveness of the SMS through: • Safety and Quality Policies • Safety Objectives • Audit & Evaluations • Analysis of Data • Corrective and Preventive Actions • Management Reviews AC 120-92 Federal Aviation Administration SL-20
- 20. SA Process Circle Diagram Continuous Monitoring: Day to day, flight by flight Job by job – supervise! Design/Re-design • Changes • Failures Back to SRM Internal/External Audits: Go out and look Analysis, Assessment, & Decision Preventive/Corrective Action: Getting back on track Affirmed: Expectations met Internal Evaluation Employee Reporting: Listen the front line Investigation: Learn from error Federal Aviation Administration SL-21
Editor's Notes
- {"16":"Data Analyses results are used for many things:\nClick\nComparing activities and performance against a criteria or objectives, such as regulations, international standards and company policies,\nClick\nCompared activities and performance against industry, international and company norms,\nClick\nPatterns from multiple data points, such as employee reports, audits, investigations,\nClick\nAnd Trends over time; be cautious using trends,\n. People often misuse the word “trends” when they mean “patterns.”\nIf you do not have data points that are comparable, they cannot be used to predict trends.\nYou must have stable, reliable measures at each time sample for a valid trend.\n","5":"The first step of Safety Assurance is to gain a fundamental understanding of the domain of the process, the system. This requires a comprehensive system description…\nClick\n…the who, what, when, where, why and how of the system, as it is documented and as it functions day-to-day. \nClick\nIt is important to understand how the system works and what tasks are involved in key processes, for example; during operation, what are the critical activities that must occur in flight operations, maintenance, ground operations, etc. The system description should include existing risk controls.\nThe operation of the system must be monitored , for example flight control; does it routinely operate smoothly, how about during weather events, multiple maintenance cancellations, how does NOTAM distribution work, is their documentation and records up to date?\nThe bottom line is, when we assess the Systems Operation, we need to ask if the situation is acceptable and are the goals and objectives being met for safety.\nSafety Assurance is linked with Safety Risk Management in the requirement to monitor risk controls during operations.\nAdditionally, if through monitoring a systemic problem is identified, SRM should be applied to that process.\nWhen risk controls are acceptable, the system is placed into operation and written documentation is in place to describe the system.\nIf risk controls are not acceptable, we need to determine whether to apply corrective action or, if the design is defective, return to the SRM process.\nBy assessing risk during system operation, the safety assurance begins its process.\n","11":"We’ve discussed Internal Audits and now will discuss Internal Evaluations….well…..what’s the difference?\nAn Internal Audit is performed by the operational divisions, on themselves. Flight operations does an internal audit on Flight Operations. In an Internal Evaluation, the divisions receive evaluations from someone outside their division. \nInternal Evaluations:\nClick\nAre accomplished by company personnel who are functionally independent of the process being evaluated,\nClick\nFocuses on the technical processes as well as the end product / service,\nClick\nReviews the results of internal audits for effective solutions, and finally\nClick\nSamples outputs of the SA components.\nThe internal evaluation function also requires auditing and evaluation of the safety management functions, policymaking, safety risk management, safety assurance, and safety promotion. These evaluations allow the management officials responsible for the SMS to inventory the processes of the SMS itself. \n","17":"Data is only meaningful to Management, if the data is in a meaningful form, and analyzed. Once data is analyzed into information, conclusions are drawn from the information to form bottom-line decisions. The conclusions and bottom-line decisions are the purview of System Assessment, the fourth step in the Safety Assurance process. \nClick\nSystem Assessment applies value judgments to the situation as understood in terms of available information and the decision-maker’s past experience. This is where questions such as “Is the situation or risk acceptable?” and “Are goals and objectives being met?” are asked and answered.\nThe primary purpose of the Safety Assurance is to assess the continued effectiveness of the risk controls put into place by the SRM process and to evaluate the operational system.\nThere are three possible options for the decision makers during system assessment: \nFirst, the objectives ARE being met, No action is taken and the system will be monitored, analyzed and assessed again sometime in the future.\nClick\nThe second option is where deviations to existing controls are discovered. Some of these failures maybe due to:\nLack of supervision,\nLack of resources,\nLack of training, or\nPoor Job aids. \nIn these cases the controls themselves appear to be okay, or maybe need minor correction, but the employees, for whatever reason, are not following their processes and procedures. The standard requires a structured, documented process for preventative and corrective action to place controls back on track. \nClick\nThe third option is where new hazards are discovered or a risk control has failed. Sometimes everyone is doing everything that is expected, but it just isn’t working to control the level of risk. Possibly the conditions have changed so that the original control is no longer is appropriate. Or it could be because of changes in contracts, changes to airports, new equipment, changing demographics of employee hiring pools or a variety of new conditions. At any rate, we’ve identified a new and uncontrolled hazard so we need to return to the SRM process to re-design the system aspects or develop new controls. \nClick\nWhen the new hazard or failed control is sent back to SRM for redesign, it will be prioritized according to its critically.\nThis Safety Assurance step is outline in the SMS Framework.\n","6":"We said earlier that Safety Assurance was designed to monitor risk controls and help with decision making. \nDecision making needs information and information comes from data.\nClick\nIt is important to collect data and do data-mining on data sources that will identify safety concerns that an organization can use to make informed decisions. There are the six types of data sources discussed in the SMS Framework.\nClick\nThey are very good data sources, each has a multitude of inputs and they have all been around for a long time.\nClick\nEach of these data and information sources exists to some degree in every organization.\nClick\nThe SMS standard formalizes requirements for each.\nClick\nSMS specifications for these data sources are left at the functional level, that is “the what needs to be done” level, allowing individual organizations to tailor them, “the how it needs to be done” level, to the scope and detail appropriate for the size and complexity of the organization.\nClick\nThere is a virtual mountain of information available to us. \nClick\nWith limited resources to review the information collected, it’s wise to consider, in advance, what are the most effective sources of data to collect. These six categories represent some of the top sources for aviation safety data within the industry.\n","12":"Another Safety Assurance data source is External Audits. \nClick\nExternal audits of the SMS may be conducted by:\nCode-share partners,\nCustomer or industry organizations,\nOther third parties, selected by the operator, or\nThe regulator, the FAA.\nThese audits not only provide a strong interface with the oversight system but also serve as a secondary assurance system.\nOrganizations may elect to have third-party audits from organizations such as IATA, CASE, ACSF or other consultants\nThere are no SMS requirements for organizations to hire an external auditor. If the company does not get audited, then there will be no external audit data. However, IF, external audit data is available, typically from code-shares or the FAA, then that data must be used in the Analysis of Data Process\n","1":"As discussed in the overview portion of this presentation, the purpose of Safety Assurance, is to evaluate the overall effectiveness of the SMS and the safety of the organization through monitoring, data tracking and analysis, and investigations. \n","18":"During the last Safety Assurance step, the Preventive/Corrective Action step, we take action on the assessment decisions we just made in the previous System Assessment step, for problem resolution. Preventive and Corrective Actions should be developed in response to deficiencies, they should be timely, provide effective resolution of the deficiency and should prevent recurrence. Remember that these deficiencies do not represent problems with the system design. Thus, a redesign of the system, using SRM, is not necessary.\nThese are examples of preventative and corrective actions…\nClick\n… we could increase supervision or emphasize procedures, fix shabby equipment, provide more training, change employee schedules or assignments and address health and wellness issues. This is not an exhaustive list.\nRemember that Management is responsible for corrective actions and for the most part the actions should reside with the operational departments cited in deficiency findings.\n","7":"What is continuous monitoring and who has the responsibility for it?\nClick\nYou probably didn’t think about SRM and SA, but it’s true, continuous monitoring is where the controls developed in SRM are directly monitored and evaluated for the Safety Assurance input, , it is the link or interface between the two. As for responsibility, the line managers have direct responsibility to:\nClick\nMonitor daily activities and processes within their area,\nClick\nAssure that processes, procedures and their controls, are complied with, and\nClick\nPeriodically assess status of risk controls to assure they function as designed\nLine organizations are the domain of technical experts in any organization and they are the most knowledgeable about the technical processes involved. \nWhy does line manager continuously monitor operational data?\nTo assess conformity with risk control,. within normal procedures, when implemented in the operational environment, \nTo measure the effectiveness of safety risk controls, \nTo assess system performance, and\nTo identify hazards\n","13":"Safety investigations part of our need for “reactive” strategies…\nClick\n…are often conducted to put the event behind us; to assure everyone that all is okay and normal activity can resume. \nClick\nHowever, under SMS, the best use of the data is to learn about system vulnerabilities, develop better strategies and prioritize resources for improved system reliability. \nConsider how ICAO defines investigation. “To prevent accidents by gathering and analyzing information, drawing conclusions, including determining causes and when appropriate making safety recommendations.”\nIf reports show a high-risk potential, then there should be a greater depth of investigation than those with low potential. In other words, the scope of the investigation must factor in the complexity of the situation. \nSMS depends, to a certain extent, on investigation, analysis of safety issues and identification of underlying hazards. The Safety Assurance Process requires an organization to collect data on incidents and accidents and use that data in the Analysis of Data Process.\nThe quality of the investigation effort is very important in order to identify the underlying hazards. Much has been written about accident investigation techniques and strategies. However, the bottom-line for SMS is that investigations must clearly identify underlying hazards. \n","2":"Safety Assurance, is the second major component of an SMS.\nBlack’s Law Dictionary…\nClick\n…gives us a definition of assurance as something that gives confidence.\nClick\nSafety Assurance…gives us confidence in our safety risk controls.\nIn other words, a properly administered Safety Assurance process provides confidence that the organization is meeting or exceeding its safety objectives by controlling risk. \nSafety Assurance helps us to control the “practical drift”, which occurs as a natural part of any organization, and gives us confidence that risk controls remain effective.\n","19":"We are finished with the five step SA process and now will explore the last two processes in Safety Assurance, Management Review and Continuous Improvement.\nManagement Review closes the SMS loop back to policy. This is where management becomes directly and personally involved in the safety management process. Through such reviews, management learns about changes in the operational environment, identified hazards and related risk controls, effectiveness of risk controls and status of corrective actions. This knowledge by management will frequently be reflected in organizational policy changes.\nClick\nManagement Reviews should be done periodically as well as during safety assessments. Specifically, management reviews should include…\nClick\n…the outputs of SRM and SA…\nClick\n…lessons learned and…\nClick\n…the need for safety changes throughout the organization.\nA complete list is outlined in the SMS Framework. \n","8":"Here are some ideas for data sources:\nClick\nThe bottom line is to first look for business purposes and then identify the information sources that an organization is already collecting to maximize the applicability and benefit available from that data source.\nFor example, a check and balance can be gained by using pay records, to determine whether a control was used – How? Well if there was an MEL item not to dispatch an aircraft without two pilots, were two pilots paid for the flight?\n","14":"The sixth information source for safety assurance is Employee Reporting. \nClick\nIt is an SMS requirement that an organization develop and implement a confidential employee safety reporting and feedback system. All employees should have a means to provide feedback to the management of the company. This is NOT just Event-Reporting. The main objective of any safety data collection and analysis system is to make events, hazards, safety trends and their contributing factors visible, understandable, and supported by useable data so that effective corrective action can be taken.\nClick\nThe system must provide confidentiality, and…\nClick\n…employees must be encouraged to use the system, Historically we find that even those organizations that have an active ASAP often do not encourage employees to raise safety concerns, report hazards, etc., they just look at events.\nClick\nData from employee reports may identify emerging hazards. Think about this, who better to see initial hazard indications than those actually accomplishing the activities.\nFrom the Human Error perspective, the behavior of individuals or groups involved in incidents or “near misses” may not differ greatly from that observed when accidents occur. Generally, the cognitive failures, problems in decision making, communication breakdowns, distractions, and all the other factors which contribute to the sum total of behavior in an accident will also be present in incidents.\nClick\nThe data collected in the safety reporting and feedback system shall be included in the Analysis of Data. \nTthere are several ways to kill an employee reporting system: Burn the reporter or Burn the data.\nThe “burn the reporter” reference means: If an employee is penalized for reporting, a rapid death of the reporting system will occur because the program immediately loses its credibility, and other employees, in turn, will not report safety concerns.\nThe “burn the data” reference means: Failure to provide feedback to the reporter. The reality is that if nothing is done with the data, employees will lose faith in the process and stop submitting reports, which can cause the slow death of reporting system.\n","3":"How does Safety Assurance control drift and the effectiveness of risk controls? The outputs of safety assurance activities feed the other components of the SMS. For example: When safety assurance activities help identify hazards, we conduct a Safety Risk Management process.. When negative trends in safety performance are identified through audits, we correct them and share the information across the organization. When compliance issues are identified, we review their impact on policy and procedures. \nFundamentally, there are three Safety Assurance functions:\nClick\nCollecting and analyzing information to determine that process requirements are continuously being met,\nClick\nAssessing the performance and effectiveness of risk controls, and\nClick\nWorking hand-in-hand with, Safety Risk Management.\nAnalyzing process requirements asks: Are we doing it?\nAssessing effectiveness asks: Is it working as we intended?\n","20":"While Management Review closes the SMS loop, Continuous Improvement ties the knot that combines the elements of the SMS into a whole.\nClick\nAs stated in the SMS Framework; “The organization will develop and maintain a process to identify the causes of sub-standard safety performance, determine the implications of substandard safety performance, and eliminate or mitigate such causes.”\nThat is a tall order, just how should we go about it?\nThe SMS Framework provides guidance, “Through the use of…\nClick\n…Safety and Quality Policies…\nClick\n…Safety Objectives…\nClick\n…Audit and evaluation results…\nClick…\n…Analysis of data…\nClick\n…Corrective and Preventive Actions…\nClick\n…and Management Reviews.\nThat covers just about everything in the SMS operational environment.\n","9":"Audits….the word alone strikes fear in most aviation professional. But, consider this; under a fully functional SMS, audits are opportunities for organizational improvement. \nThe key point to remember is that Internal audits are not intended to be a test that is administered by an outside entity to be passed or failed.\nInternal audits simply take the pulse of the operation…\nClick\n…it is the Process Owner’s look at their process for information they can use to improve and keep their process on track by periodically looking at current, recent and past activities and results. Through internal audits, line managers can get data to use in making decisions and controlling operations in their area of responsibility.\nWith that in mind, let’s look at these opportunities for improvement.\nClick\nWith line managers of operating departments is where the primary responsibility for internal audits rests. With those who own the technical processes.\nClick\nThis is where hazards are encountered most directly and where deficiencies in processes contribute to risk…\nClick\n…Where audits provide direct detailed feedback to the process owners, and…\nClick\n…where direct supervisory control and resource allocation can mitigate the risk to acceptable levels.\n","15":"The overall purpose of Safety Assurance is to evaluate the overall effectiveness of risk controls and the SMS. This is accomplished through monitoring, data tracking, data analysis, and assessments. \nThe third step in the Safety Assurance process is data analysis. \nClick\nData analysis is a human activity that is used to make inferences based on data, to make sense of it, to make it understandable; in short to turn data into usable information.\nThe intent of data analysis is to look for patterns, as well as events. Data to be analyzed should be considered across all data sources such as employee reports, investigations, deviations, audits, etc.\nThe main objective of safety data collection and analysis is to make events, hazards, safety trends, and their contributing factors visible, understandable, and supported by useable data so that effective corrective action can be taken. Do not collect data that cannot be used, just because it is easy to collect.\nClick\nThe SMS is about decision-making, thus data needs to be useful in the context of making safety decisions.\nData is analyzed for the purpose of assuring the effectiveness of the SMS and of the risk controls that are in place for the operational processes in the organization. \nThe outputs of Safety Assurance activities feed other components of the SMS, thus it is important for data analysis to be valid. \n","4":"Safety Assurance is often confused with Quality Assurance; in truth, they both use many of the same principles and practices. \nClick\n• Quality Assurance focuses on continuously controlling product conformity and customer satisfaction, such as meeting production goals,\nClick\n• Safety Assurance functions ensure that risk controls perform in a way that continue to meet their safety objectives,\nClick\n• SMS encourages the integration of both systems for mutual advantage..\n","21":"Here is another way to view the Safety Assurance Processes. \nClick\nSafety Assurance is all about: Analysis, Assessment, and making safety Decisions, based on information an organization collects and evaluates.\nClick\nContinuous Monitoring: Day to day, flight by flight Job by job – This is supervision!\nClick\nInternal Audit: The next step is to Go out and look – actively conducting audits, evaluations, surveys and reviews.\nClick\nEmployee Reporting: Ensure there’s a formal and effective means to Listen the front line, and gain the benefit of the knowledge and experience of those actually doing the work.\nClick\nInvestigation: Although reactive, it is important to Learn from errors, mishaps, occurrences, incidents and accidents.\nClick\n“Affirmed” relates to analyzing whether expectations, goals, and objectives are being met.\nClick\nCorrective Action: If expectations are not being met, and the issue is not systemic, corrective actions should be used to Get back on track. This needn’t entail the same level of detail that we used in design or redesign, many times, the corrective action needed is straightforward. \nClick\nIf there are systemic issues, or there are changes to the system, or a control failure, the operation should be taken back to the Safety Risk Management Process.\nClick\nFinally, Internal Evaluations are used to evaluate the entire Safety Assurance Process, and the SMS, to ensure that they are functioning, as desired.\n","10":"What are some of the key design points of internal audits?\nClick\nFirst, they are performed by each operational department, on themselves,\nClick\nThe Director and Line managers of operational departments have direct responsibility for ensuring processes in their areas of responsibility function as designed, internal audits are part of that responsibility.\nClick\nAudits should be scheduled on a regular basis. Remember that an audit is a snapshot showing what is occurring at a particular point in time. One audit is not enough – the audit tool needs to be used regularly to be useful.\nClick\nThe audit obligation extends to subcontractors that may used to accomplish organizational functions.\nClick\nAudits should:\nDetermine conformity and performance of safety risk controls and the departments’ performance in reaching safety objectives.\nFinally remember that an audit does not make you safe. It only shows the situation during one particular period in your organization’s life. It is the action resulting from deficiencies that help to control risk and ultimately make your organization a safer place to work.\n"}