Leading Indicators for ES
Upcoming SlideShare
Loading in...5
×
 

Leading Indicators for ES

on

  • 3,489 views

 

Statistics

Views

Total Views
3,489
Views on SlideShare
3,489
Embed Views
0

Actions

Likes
1
Downloads
116
Comments
1

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Good presentation
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • (Soon to be in the new Office of Nuclear Safety and Environmental Policy, HS-21) I have been involved in identifying and generating ES&H performance measures for several years. Over this time, I have been particularly interested in how we can identify those elusive “leading indicators” we all keep searching for Through my readings and discussion, I’ve come to some conclusions about “best practices,” which I want to share today I hope that you will find them stimulating and persuasive. But I should add that these are my personal view. I’m not speaking for EH, (or HSS) or for “headquarters”
  • My intent is not to survey the specific performance measures used in these different examples, but rather to step back and identify a key organizing principle, and to cite a few examples along the way.
  • Grace Wever worked for Kodak at the time her book was written, and was active with the Great Lake Council of Industries. She later moved to Peat, Marwick.
  • Wever adds a third level (additional leading indicators). These reflect fuller integration of ES&H into corporate management, and fuller integration with stakeholders and customers. I’m not going to address this third level of performance measurement in any detail. (Although I do think that some DOE sites and contractors are currently doing things that fall within level 3.) But it’s important to be reminded that there are higher levels of excellence to which we can aspire. Transition: Remind ourselves why we are on this search for the ‘holy grail’ of ‘leading indicators’??
  • Transition: The European Process Safety Center makes this point explicitly
  • DOE sites certainly include ‘operations where there may be potential for severe accidents’. Transition: The British Health and Safety Executive makes much the same point
  • This list encompasses a diverse collection of “best practice” approaches. None of these is directly transferable to DOE. But each has useful elements to consider Its not my intent to describe any of them completely. Rather, I will try to highlight some important similarities and differences, and then make some summary observations. References are listed at the end of the presentation, so that you can explore any of these more fully. All of these different “best practice” approaches have a common theme: the need to measure how key elements of the management system are functioning.
  • The first “best practices” example I want to discuss is an excellent guide produced by the British health and Safety Executive.
  • A broad definition of performance measurement, and of its purpose.
  • Health and Safety Executive takes a broad systems approach. For example, they explicitly monitor -- not just identify, but track -- what they call ‘inputs’ that is, the hazards faced or created by the organization’s activities. And they monitor the elements of the H&S management system And they measure ‘outcomes’ or ‘lagging indicators.’
  • H&SE identifies five key elements of a H&S management system. These look very much like “Plan – Do – Check – Act.” Or like ISO 14001. Or like Integrated Safety Management. Within the ‘process’ part of framework, the performance measurement system must cover each element of this management system.
  • Measurement, in this approach, is not as easy as collecting outcome (lagging) data. (And I acknowledge that this has its challenges as well.) It’s a qualitatively different, and much more wide-ranging, activity. It takes advantage of a broader range of information, and systematically seeks it out.
  • Transition: The next “best practice” system I will discuss is the Total Quality Environmental Matrix, developed by the Great Lake Council of Industries.
  • Transition: In Australia, the government occupational safety and health office is promoting the use of “positive performance indicators”
  • Note that ‘inputs’ has a different meaning here than in the British system.
  • Transition: At the ISM meeting in April, Kurt Krueger from GE described GE’s corporate Health and Safety Framework.
  • Computerized reporting and
  • CCPS was formed in 1985 by the American Institute of Chemical Engineers to promote the improvement of chemical process safety Also rolls up into score for each element
  • Under each element, you can click down to detailed questions, definitions, and guidance.
  • DOE has many pieces in place which could support a system such as this. We don’t consistently use a management system framework to organize and pull all this information together Office of ES&H Evaluations in the Office of Security and Safety Performance Assurance Moving to the new Office of Health, Safety and Security as HS-64)
  • Office of Independent Evaluations (SP-44) recently published the report of their inspection of the Savannah River Site The Inspector General’s office recently issued a report on the Corrective Action Program at the Office of Civilian Radioactive Waste Management
  • As an example, the inspection at Savannah River Site (report issued February 2006) Looked at four SRS activities Tritium facilities H-canyon operations H-canyon maintenance and construction SRNL and assessed them against the five ISM core functions Also looked at five elements related to Essential System Functionality Configuration management programs and supporting processes Engineering design and authorization basis Surveillance and testing Maintenance and improvement Operations Office of Independent Oversight rates each as Effective Performance Needs Improvement Significant Weakness Transition: If we look at 17 inspections, over three years, we begin to see a pattern
  • I tabulated these last year, in connection with the Department’s Compliance Improvement Initiative. This summarizes the ratings given by the Office of Independent Oversight in these four areas in the 17 ES&H inspections from 2002 through 2004.
  • Many of these will be qualitative measures.
  • Patty Bubar, in the next room, is describing a proposal for a new set of DOE Corporate Performance Indicators. These have the potential to improve our use of precursor information which we already collect through ORPS and other systems. The broader vision I’m talking about today is not inconsistent with this proposal. This vision generalizes from a variety of “best practices” from around the world to identify a broader conceptual framework, within which we can make use of a wider range of information to assess how well we are really managing risk. It is a framework, or a mindset, which can inform our improvement efforts over the coming years.

Leading Indicators for ES Leading Indicators for ES Presentation Transcript

  • Steven R. Woodbury DOE Office of Nuclear Safety and Environmental Policy Leading Indicators for ES&H Performance: Measuring How Well Our Management System Is Performing ISM Best Practices Workshop September 12, 2006
  • Overview
    • The mix of ES&H performance measures changes as an ES&H program matures.
    • DOE currently uses many lagging measures.
    • Our ‘next level’ of performance measurement should focus on how well key elements of our ES&H management system are performing.
    • There are instructive examples in the literature of best practices on which we can build.
    • But nobody says this is easy.
  • Three Levels of Performance Measurement
    • Grace Wever identifies three types of performance measures that organizations may employ:
      • 1. Operational indicators (lagging)
      • For example:
        • waste generation
        • accident/injury rates
        • regulatory violations
        • 2
      • 2. Identifying underlying management system deficiencies (leading) For example, the quality of:
        • training
        • audits
        • communications
        • documentation
        • review processes
  • Three Levels of Performance Measurement (continued)
    • 3. Identifying gaps in internal competencies and external relationships (leading) For example,
      • Organization begins to measure:
        • Impacts of products, operations, or services on public health and the environment
        • EHS research and development
        • Capital investments
        • Stakeholder and customer partnerships
      • Organization strives to link financial, human resource, operation, EHS, and other goals and measures
    • “ The mix of an organization's performance indicators tends to reflect the maturity of its EHS program.”
    • Grace Wever, Strategic Environmental Management , pp 158-9
  • Importance of Leading Indicators
    • The importance of developing good leading indicators goes beyond “It’s hard to steer by looking in the rearview mirror.”
    • The importance goes beyond continuous improvement “We should do what leading organizations are doing.”
    • We need to be concerned because even good performance, on good lagging indicators, can’t assure us that risks are really being controlled.
  • Outcome Measures are Not Sufficient
    • “When safety is good and injury and loss rates are low, then [outcome] measurements are not sufficient to provide adequate feedback for managing safety.
    • For operations where there may be potential for severe accidents, the likelihood of such events must be extremely low. This means that the absence of very unlikely events is not, of itself, a sufficient indicator of good safety management. ”
    • --European Process Safety Center, 1996, p. 3.
  • Outcome Measures are Not Sufficient
    • “Health and safety differs from many areas measured by managers because success results in the absence of an outcome (injuries or ill health) rather than a presence. But a low injury or ill-health rate, even over a period of years, is no guarantee that risks are being controlled and will not lead to injuries or ill health in the future. This is particularly true in organizations where there is a low probability of accidents but where major hazards are present. ”
    • – A Guide to Measuring Health & Safety Performance Health & Safety Executive (UK), p 5
  • Some Best Practices in “Next Level” Performance Measurement
    • Guide to Measuring Health & Safety Performance (British Health and Safety Executive)
    • Total Quality Environmental Management Matrix (Grace Wever, Council of Great Lakes Industries)
    • Positive Performance Indicators (Australia, Department of Employment and Workplace Relations)
    • General Electric Company
    • ProSmart software (Chemical Process Safety Council)
  • Health & Safety Executive (UK)
    • Guide discusses performance measurement in the context of a health and safety management system
    • Discusses the limitations of traditional “lagging” measures
    • Provides a systematic approach to deriving measures which link to the H&S management system
  • Why Measure Performance?
    • “ The primary purpose of measuring H&S performance is to provide information on the progress and current status of the strategies, processes and activities used by an organization to control risks to health and safety”
    • – A Guide to Measuring Health & Safety Performance Health & Safety Executive (UK), p 7
  • HSE Guide – a Balanced Approach
    • Performance measurement should be based on a balanced approach, which addresses:
    • Input
    • Monitoring the scale, nature and distribution of hazards created by the organization’s activities
    • Process
    • Active monitoring of the adequacy, development, implementation and deployment of the H&S management system
    • Outcomes
    • Reactive monitoring of adverse outcomes resulting in injuries, ill health, loss and accidents with the potential to cause injuries ill health or loss (‘measures of failures’)
  • HSE Guide – Measure the Management System
    • The H&S management system is the process which turns uncontrolled hazards to controlled risks
    • The key elements of
      • Policy
      • Organizing
      • Planning and implementation
      • Measuring performance
      • Audit and review
    • all need to be in place to control risks effectively
    • The performance measurement system must cover each element of the H&S management system
  • HSE Guide – What and How to Measure
    • Measuring elements of the H&S management system should cover three aspects:
      • Capability
      • Implementation
      • Deployment
    • The measurement process can gather information through:
      • Direct observation of conditions and people’s behavior
      • Talking to people to elicit facts and their experiences as well as gauging their views and opinions
      • Examining written reports, documents and records
  • HSE Guide – Observations
    • Performance measurement is approached in the context of the health and safety management system
    • Guide acknowledges the need to measure outcomes – b ut this is not sufficient:
      • ‘ measures of failure’
      • doesn’t guarantee that risks are really controlled
    • Approach is not limited to Health and Safety – completely applicable to management across ES&H
    • Focus is on the performance of key management system elements
    • Measures are largely qualitative
    • Provides a clear framework for thinking about ES&H management and performance measurement
  • Total Quality Environmental Management Matrix
    • TQEM identifies 7 performance categories, based on Baldrige Criteria most relevant to EHS management, as well as the ISO 14001 standard
    • Identifies performance levels (1 – 10) (based on descriptive text)
    • Detailed self-assessment questionnaire is provided to support evaluation.
    • Within a category, one cannot score at a ‘higher’ level unless performance is fully satisfactory at all ‘lower’ levels
    • Developed in the 1990’s by Grace Wever, and the Council of Great Lakes Industries
    Seven Categories Ten Performance Levels
  • Total Quality Environmental Management Matrix – Categories
    • Seven categories (different weights)
    • Leadership (15%)
    • Information and Analysis (7.5%)
    • Strategic Planning (7.5%)
    • Human Resource Development (10%)
    • Process Management (15%)
    • Environment, Health & Safety Results (30%)
    • Customer/Stakeholder Satisfaction (15%)
  • TQEM Performance Levels – Environmental Health and Safety Results
    • Example: ‘ES&H Results’ Category
    • 10. Benchmarking shows unit is ‘best-in-class’
    • 7. EHS improvements contribute to financial and business improvements
    • 5. Positive improvement trends in units’ key EHS measures
    • 4. EHS measures reviewed and improved at least annually
    • 3. EHS results compared with objectives and targets; used to improve effectiveness of management systems and performance
    • 1. EHS performance measures identified; baseline data and trends collected
    EH&S Results
  • TQEM Performance Levels – Strategic Planning
    • Example: ‘Strategic Planning’ Category
    • 9. Information on competitors’ EHS strategies used to improve units’ strategies and performance
    • 8. EHS integrated into long and short-term business plans for all of unit’s products, processes and services.
    • 6. EHS plans and deployment consistently aligned at all levels of the unit
    • 4. Long and short-term plans that include EHS objectives are reviewed and improved at least annually. Key measures include EHS measures
    • 1. Long-term and short-term planning process used that addresses EHS needs; annual operating plan includes EHS management needs
    Strategic Planning
  • TQEM Performance Levels – Human Resource Development
    • Example: ‘HR Development’ Category
    • 9. EHS needs fully integrated into unit’s HR development plan. Training/education for EHS staff include key business knowledge
    • 7. Employees proactively initiate activities to improve EHS performance
    • 5. Measures of EHS training effectiveness in place; employees with potential to impact EHS are competent to perform EHS responsibilities
    • 3. All employees have received appropriate EHS training; employees aware of EHS compliance requirements.
    • 1. EHS training needs identified, resources committed.
    Human Resource Development
  • TQEM Matrix – Observations
    • Not limited to environment – completely applicable to management across ES&H
    • System is based on explicit, but qualitative, criteria for ES&H management
    • Focus is on qualitative measures of performance in key management categories
    • Published matrix is fully defined (not open-ended)
    • Matrix was developed more than 10 years ago – it should be updated on the basis of additional experience
    • Its importance is as a conceptual framework; the specific content can be adapted
  • Positive Performance Indicators
    • ‘ Positive Performance Indicators’ (PPIs) focus on assessing how successfully an organization is performing by monitoring the processes that provide good OHS outcomes.
    • PPIs are to be developed in addition to traditional ‘outcome’ indicators.
    • PPI program was developed by the Australian Government’s Department of Employment and Workplace Relations
  • Positive Performance Indicators – Based on Quality Management
    • PPI is based on a Quality Management approach. PPIs may be
    • Inputs (Key Activities)
      • Measures of actions or initiatives undertaken to improve OHS
      • Provide indicators of commitment and effort, but not of effectiveness
      • Can provide useful information on participation, leadership and communication
    • Processes (Monitoring Key Risks)
      • Identify key risks
      • Identify key contributors to outcomes of concern
      • Develop measures to monitor behaviors and practices
    • Outputs (Milestones)
      • Achievement of objectives
      • Progress toward achievement of higher-level OHS goals and targets
  • Positive Performance Indicators – Part of a Management System
    • PPIs are only useful if an organization has a systematic approach to management of OHS
    • Many variations exist, but all have the following principles:
      • Commitment and policy
      • Planning
      • Implementation
      • Measurement and evaluation
      • Review and improvement
  • Positive Performance Indicators – Categories of PPIs
    • Commitment and policy
      • Measures demonstrated commitment to improve OHS performance
    • Planning
      • Measures what procedures are in place to eliminate workplace injury and disease
    • Implementation
      • Measures the capability and support mechanisms that are necessary to achieve OHS objectives and targets
    • Measurement and Evaluation
      • Measures the extent to which workplace health and safety is monitored and evaluated so that issues can be identified and corrective action taken
    • Review and improvement
      • Measures the effectiveness of the OHS management system, and its continuing suitability
  • Examples of PPIs – Commitment and Policy
    • Examination of records
    • Employee questionnaire/survey
    • Evidence of OHS policy statement signed by CEO
    • Frequency and quality of OHS reporting by or to Senior Management
    • Senior managers’ performance appraisals include OHS
    • Percentage of workforce and contractors covered by consultation processes and OHS representation
    • Rating of effectiveness of employee participation on OHS management
    How to Measure Performance Indicators
  • Examples of PPIs – Implementation
    • Observation – walk through inspections/ audits
    • Examination of hazard reports
    • Examination of hazard logs
    • Review of maintenance log
    • Analysis of accident and incident reports
    • Percentage of workplace inspections conducted over a specified timeframe
    • Percentage of high risks identified over a specified timeframe
    • Proportion of items identified through safety walks and inspections that are repeat items
    • Proportion of reported incidents that do not result in injury compared with those that do
    • Percentage of planned management visits conducted over a specified timeframe
    • Percentage of managers and employees that have received OHS training (e.g. induction, job-specific, hazard management, emergency procedures….)
    How to Measure Performance Indicators
  • Positive Performance Indicators – Observations
    • PPIs are not limited to Occupational Health and Safety – completely applicable to management across ES&H
    • PPIs focus on identifying key elements of the ES&H management system, and measuring how well these key elements are functioning
    • PPIs are non-prescriptive and open-ended: approach provides great flexibility at the facility and site level
    • Most PPIs are qualitative, based on assessments, record reviews, surveys, or audits
    • The Guidance document assumes a fairly basic management system. DOE sites generally have more sophisticated ES&H management systems in place, with the opportunity to develop more sophisticated measures.
  • General Electric – Health and Safety Framework
    • At the April ISM meeting in Albuquerque, Kurt Krueger described GE’s corporate-wide Health and Safety program
    • “ One standard, one program, one set of metrics for every GE facility around the world – No exceptions!”
    • GE’s Health and Safety Framework includes a mix of 21
      • Management system elements For example:
        • Training
        • Accident investigation and follow-up
      • Subject matter areas For example:
        • Industrial hygiene
        • Chemical management
        • Motor vehicle safety
  • General Electric (continued)
    • Detailed questions
    • GE-specific guidance & expectations
  • General Electric (continued)
    • Scorecard ratings are based on
      • Plant self-assessment (540 questions, twice a year)
      • Periodic corporate audits
    • Tied to monthly site operational metrics and supervisor scorecards,
      • “ tailored to site operations”
      • “ designed to drive supervisor behaviors that will find and fix H&S issues before an accident finds them”
    • GE has a parallel system for assessing environmental performance
    • GE also continues to track “trailing metrics” For example:
      • Recordable injury and illness rates
      • Lost time injury and illness rates
      • Permit exceedances
      • Notices of noncompliance
  • General Electric – Observations
    • Integrated into GE’s broader management system
    • Reflects a balance between frequent site self-assessment and reporting, and periodic headquarters audits
    • GE chose to implement a uniform corporate system. While this might not be DOE’s choice, there is still a lot to learn from the overall framework, and from the measures GE’s chose to include
  • ProSmart – Center for Chemical Process Safety
    • In 1993, CCPS began a project to measure the effectiveness of process safety management systems
    • Their objective was “to provide management with the tools for assessing the health of process safety management systems on a real-time basis”
    • Based around 12 management system elements
    • Extensive set of computer-based questions, using qualitative and quantitative measures
    • Rolls up responses into a single score or ‘index’
  • CCPS ProSmart – Based on 12 Management System Elements
    • CCPS identified 12 key management system elements, such as
      • Accountability
      • Process knowledge and documentation
      • Capital project hazard review
      • Training and performance
      • Management of change
      • Incident investigation
      • Audits and corrective action
    • ProSmart generates a score, which is continually updated
  • CCPS ProSmart – Based on 12 Management System Elements
  • CCPS ProSmart – Drills Down to Detailed Evaluation Questions
  • ProSmart – Observations
    • Derived from a quality management approach
    • Focus is on specific management system elements
    • Established for chemical process safety, but in principle could encompass environment, safety and health
    • Like TQEM and GE, ProSmart provides a set of detailed assessment questions, for qualitative assessment against established criteria
  • Key Aspects of “Next Level” Performance Measures
    • “Next Level” performance measures identify and measure how well key elements of the ES&H management system are functioning
    • Measures tend to involve
      • Precursor analysis and trending
      • Observations, assessments and audits
    • Measures tend to be qualitative rather than just counting events
    • (This doesn’t mean they can’t be rigorous and replicable)
  • DOE Has Many Pieces in Place
    • Analysis and trending
      • Data collection
      • Trending and analysis
      • Root cause analysis
    • Observations, assessments, audits
      • Contractor self-assessment processes
      • DOE oversight activities
      • Annual ISM assessments should systematically address management system elements
      • Inspections by the Office of ES&H Inspections (HS-64, formerly SP-44) address ES&H management system functioning at a high level (ISM core functions)
      • Individual IG audits address some management system elements
  • Audits and Inspections
  • ES&H Inspections
    • (colors added)
    • Define Scope of Work
    • Analyze the Hazards
    • Identify and Implement Controls
    • Perform Work Within Controls
    • Feedback and Continuous Improvement
    • (contractor) (DOE)
    • Essential System Functionality
  • ES&H Evaluations of 17 DOE Sites 2002-2004   15 2 Feedback and Improvement   5 12 Work within Controls 1 11 5 Develop and Implement Controls   7 10 Analyze the Hazards Significant Weakness Needs Improvement Effective Performance Management System Area
  • Summary
    • There is a common thread of best practice, across many different organizations: to move beyond outcome (lagging) indicators, we need to start systematically measuring how well key elements of our management system are performing.
    • This is not as easy as counting occurrences (injuries, enforcement actions): it must involve walk-throughs, audits, surveys, and other varied approaches.
    • This ‘next level’ of leading performance measurement complements, and does not replace, outcome (lagging) measures.
    • Approaches developed for process safety, worker safety, environment, can be applied across ES&H. We need to be eclectic and ecumenical in our thinking.
    • Nobody has a turnkey system that’s right for DOE. We need to identify and build on the best of these various approaches.
    • Nobody says it will be easy.
  • Next Steps
    • Identify key ES&H management system elements
      • We currently categorized some information (e.g., ORPS reports; Independent Oversight Assessments) around ISM’s five broad core functions
      • Within these core functions, we can identify more specific management system elements, including:
        • Identification of requirements - Corrective action management
        • Hazard identification - Lessons Learned sharing
        • Communication - Performance Measurement
        • Training - Worker involvement
        • Documentation - Audits and assessments
        • Periodic top management review
  • Next Steps (continued)
    • Identify the various things we are already doing to assess how well these elements are functioning
    • Identify and fill gaps as appropriate
    • Organize and present the information in a way which provides current feedback to management on how effectively key management system elements are functioning at each site and each facility
    • In Sum: Use the health of these key management system elements as the conceptual and organizing framework to strengthen our ability to know how well we are managing risks
  • In Closing …
    • “ Far better an approximate answer to the right question which may be difficult to frame,
    • than an exact answer to the wrong question which is always easy to ask.”
    • – statistician John W. Tukey
  • References
    • Grace Wever, 1996, Strategic Environmental Management: Using TQEM and ISO 14000 for Competitive Advantage , John Wiley and Sons, NY
    • European Process Safety Centre (Jacques van Steen, ed.), 1996, Safety Performance Measurement , Institution of Chemical Engineers, Rugby (UK) (distr by Gulf publishing Co., PO Box 2608, Houston, TX 77252-2608)
    • Health and Safety Executive (United Kingdom), 2001, A Guide to Measuring Health & Safety Performance http:// www.hse.gov.uk/opsunit/perfmeas.pdf
    • Commonwealth of Australia (Department of Employment and Workplace Relations), 2005, Guidance on the Use of Positive Performance Indicators to Improve Workplace Health and Safety http://www.nohsc.gov.au/PDF/Standards/GuidanceNotes/ASCCPPIGuidanceBooklet.pdf
    • Kurt Krueger, “Creating a Culture of Safety Excellence: The Journey and the Prize,” talk at DOE ISM Champions Workshop, Albuquerque NM, April 2006 http://www.eh.doe.gov/ism/workshops/Creating_a_Culture_of_Safety_Excellence.pdf
    • George Eckes, The Six Sigma Revolution: How GE and Others Turned Process into Profits , New York: John Wiley and Sons, 2000.
    • Center for Chemical Process Safety, ProSmart software http://www.aiche.org/CCPS/Publications/Software/ProSmart/index.aspx
  • Contact Information:
    • Steven R. Woodbury
    • DOE Office of Nuclear Safety and Environmental Policy
    • 202-586-4371
    • [email_address]