Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Evaluating Out Of Tolerance Instruments Webinar
Similar to Evaluating Out Of Tolerance Instruments Webinar (20)
More from Transcat
More from Transcat (20)
Recently uploaded
Recently uploaded (20)
Evaluating Out Of Tolerance Instruments Webinar
- 1. Webinar January 21, 2015 Presenter: Phil Mistretta, Metrology Manager phil.mistretta@transcat.com As-Found: Out-of-Tolerance …. …now what do I do?
- 2. What’s the Big Deal about Out-of-Tolerance Equipment? • There is additional risk to all products on which it was used • The potential risk can lead to dangerous situations and additional business costs • So where do you turn for guidance?
- 3. Objectives 1. Practical guidance on evaluating Out-of- Tolerance conditions through a systematic approach to help concentrate your efforts to identify those areas that will need further analysis. 2. Key elements in developing a Measurement Assurance Program which will make OOT evaluations quicker and easier Bonus Good Metrology Practice (GMetP) tips
- 4. Calibration • Lets define a few things …… • Calibration is ... a comparison to a metrology laboratory standard of known performance and documented uncertainty, to the unknown behavior of a unit submitted for calibration • When it fails expected performance, non- compliant, Out-of-Tolerance (OOT)
- 5. Traceability
- 7. Out-of-Tolerance Investigation 1. What – Review the calibration certificate in detail 2. When – Determine the time frame that needs to be examined 3. Where – Determine where the instrument was used 4. How – Determine what the instrument was to measure 5. Mechanics / mathematics of the evaluation
- 8. What • What Functions / Parameters? • What Range? • What test point? • How many OOT Test Points? • Was the entire Range OOT? • Was the a linearity issue? • Was it just the Zero point? • Was it just the Full Scale point? • Are there any test point at the limits? • Are there any test point just in-side the limits?
- 9. What - Example Range: -10°C to 160°C Accuracy: ± 0.06°C OOT: 2 points, 100°C and 150°C
- 10. When • What time frame do I need to investigate? • When was it last KNOWN to be in-tolerance? • Is there any measurement data to support a shorter time frame? • Are there any Preventative Maintenance documents? • GMetP Tip • Document, document, document • Process control charts • Mid-cycle checks, inter-comparisons, cross-checks • Calibration Interval Change Policy • ……..Document
- 11. When - Example Last know In-Tolerance Date: As-Left date of Previous Calibration From: January 5, 2014 To : January 15, 2015 GMetP Tip Document any time the unit is removed from service
- 12. Where • Where was the unit used during the time frame? • What department? • What area? • What product line? • GMetP Tip • Area-ID-Location • Rack mounted equipment does not walk • 5S / Visual Management – shadow boards • Tool Cribs – Sign-Out, Sign-In • Electronic tracking, Bar codes, RF ID tags, etc • Work Station or Bench assignments
- 13. Where - Example • Where: the unit was used in the Thermometry Lab Only
- 14. How During the time frame in question, in all locations where it was used, for the OOT Parameters, Ranges Test Points… • Check end user procedures? • Check work Instructions? • Check Engineering Specifications? • Check Process Documentation? • Check All Revision of the documents?
- 15. How - Example • This unit is called out by Calibration Procedure CP-0303 R3, and CP-0310 R0 • CP-0303 R3 states this unit is used to monitor a 25°C Bath • CP-0303 R3 is dated 1/1/2010 • CP-0310 R0 states this unit is used to monitor a 100°C Bath
- 16. Analysis • We know: – What – When – Where – How • Assessment of error – did the error impact product? • Magnitude and Direction of error • Error Impact on Product
- 17. Analysis - Example We know: – What: All reading above 100°C were OOT – When: 5 Jan 2014 to 15 Jan 15 – Where: Thermometry Lab Only – How: 100°C Bath Magnitude : 0.04°C Direction: Thermometer read too high
- 18. Analysis - Example We need to know what the product would have read at 100°C • Since the UUT was reading too high, all products readings will also be too high, by this amount • Since all reported product readings were too high, we must subtract the UUT’s error from the Products reported readings to get the products actual behavior at 100°C • Now check the corrected Product indication against the Process tolerances • Did it still Pass? • Did it now Fail? • Repeat for all product meeting our investigation criterion.
- 19. Measurement Assurance Programs 1. R&D Product Specifications 2. Manufacturing Process & Documentation - Tip: Identify Critical Measurements - Tip: Set Warning Tolerances 3. Selection of Suitable Test Equipment – Tip: Select single parameter vs. multifunction – Tip: Smart Accuracy Ratios 4. Quality Control Program - Tip: Process control charts for critical measurement - Tip: Mid-cycle checks, cross checks 5. Preventative Maintenance Program – a) Regular Calibration – b) Calibration with As-found / As-Left & Uncertainties – c) Calibration Interval Adjustment Policy
- 20. Summary • Out-of-Tolerance Statements • OOT Investigation • What • Where • When • How • Mechanics and Mathematics • Measurement Assurance Program Elements • R&D • Equipment Selection • Process & Documentation • Quality Control • Preventative Maintenance • Equipment Control • Product Traceability
- 21. Phil Mistretta, Metrology Manager phil.mistretta@transcat.com Questions? Thank You
Editor's Notes
- Welcome
- What the Big Deal about Out-of-Tolerance Equipment? When calibrated test equipment is found with an out-of-tolerance condition, there is additional risk to all products on which it was used. It is important to understand the magnitude of the potential risk because it can lead to dangerous situations and additional business costs. Business Costs – For example everyone has heard about the huge air bag recall of the Takata Corporation, they are forecasting a 218 Million dollar loss this fiscal year along with responsibility fo at least 4 deaths and 139 injuries Here at Transcat we calibrate and process over 100.000 customer unit each year and roughly 10-15% of those unit are found OOT. We have over 8000 calibration standards and assets and you can bet some of them are found OOT, just like every calibration lab out there. Back in 2011 a few of my coworkers Howard Zion Director of Technical Affairs, and Jeremy Sims our Quality Manager) and I were batting around ideas for white papers and this topic came up and we all thought it was a great idea, so I researched it and I pretty much found nothing out there. We also ran it by one of our customer Bob Sutton from Terumo Cardio Vascular Systems who thought it was a great topic., so I wrote a paper on this topic and presented it at the annual NCSLI conference. (The full paper is available through the Transcat website.) After reading the 11 page paper Bob Sutton called me up and told me, great start Phil, but I want more, you have to do more. Well Bob got what he wanted, we put together a 1 day, on-site, hands on seminar on Evaluating OOT units. Additional information is available the Transcat Consulting Service.
- Objective of Webinar There are just 2 things I hope you take away from todays webinar 1 Practical – OOT investigations aren’t any fun, I’ve been in Metrology for over 25 years and I’ld do almost anything to prevent from having to do another one, which leads us to #2 2 Key elements for a MAP CLICK Along the way I’ll also give a few tips-from the trade
- Lets define a few things first so we are all on the same page….. Believe it or not, , Metrologists have their own dictionary call the Vocabulary of International Metrology (VIM) it has a page long definition that I’m not going to cover, but in it simplest form: Calibration is a comparison to a metrology laboratory standard of known performance and uncertainty, to the unknown behavior of a unit submitted for calibration. Non-compliance What does out-of-tolerance mean? When the unit under test (UUT) does not meet the expected test limits, it is considered to be Out-of-Tolerance.
- Traceability is one of those buzz words scattered about in standard and on certificates BIPM – Define SI Units NMI – Realize SI Units, create the most accurate representations of the SI & Primary Labs –Disseminate those measurements across the world Secondary Labs – Calibration Services to End Users Equipment Applications – users of test equipment Consumers - Each level is linked to the one above it through Calibration Documentation to make a complete unbroken chain So remember when I said We have over 8000 calibrated assets and you can bet some of them are found OOT. For example our Primary Stands Lab in Houston cals all our Multifunction Calibrators, if there is an OOT there it could affect all 100 of our standards Which in turn could affect tens of thousands of Customer Units run across those calibrator, Which in turn could affect countless end products or services So its kind of a Big Deal
- I can tell you from experience as a Lean Manufacturing Engineer responsible for 4 production cells, with the calibration program as an additional duty to manage nearly 3000 calibrated items, that AS-Found statement was the most valuable piece of information on my calibration certificates. What does the Statement of compliance mean? The statement As-Found: In-tolerance is a statement of compliance. It means that the entire instrument, all functions, parameters, ranges and test points - are within the calibration specifications at the time of calibration, for the stated conditions at the location where the calibration took place. in essence the metrology laboratory, staffed with measurement experts, has completed an initial data evaluation of the data (not all labs will provide this info) Now As-Found: Out-Of-Tolerance (OOT) condition indicates that at least one data point in the data report drifted or shifted beyond the allowable tolerance limits which means the measurements it was most recently providing may not have been accurate. In an OOT situation the laboratory measurement experts have indicated that this unit had a problem and needs further analysis by the customer. The OOT statement of compliance is the trigger in the quality system to start an investigation.
- The Investigation Process The object of the OOT investigation process is to identify if there are any at risk products due to the OOT instrument. This process is designed to eliminate products without risk and to narrow down the pool of at risk products if any. This is an investigation; caution must be observed against having the end result already in mind. It is tempting to want the conclusion to show that there were no at risk products. Don’t start with a foregone conclusion, the investigation process will lead you to the appropriate conclusion much like the evidence a crime scene investigator follows leads to the guilty party WHAT, WHEN, WHERE, HOW Yep, that all there is to this, 4 questions and some math. I hope I don’t loose anybody right now because this looks so simple because the devil is in the detail folks. The sequence also matters
- What was Out-of-Tolerance? The first thing to do is to read through the calibration data to get a firm understanding of what specifically failed calibration. Identify what functions, parameters, ranges and test points were found out-of-tolerance. The ideas is to answer questions like: The quality of the calibration and quantity of data available can have a tremendous impact on narrowing the scope of the investigation at this point. This is why you need a full calibration data , as found and as left data and uncertainties.
- Tip – Make sure you are looking at the AS FOUND DATA. Sometimes you get 2 separate reports
- When did it happen? The next step should be to identify the time frame during which questionable measurements may have been taken. This objective is to identify a specific time when the instrument was last known to be taking correct measurements. Often, this will be As-Left measurement data on the previous certificate. This will provide a starting point from which to work. ! Any effort you can make to limit the time frame in question will save you a lot of later in the Investigation and Analysis Phases. It’s much better to only have to look at 6 months of data or 3 months CLICK Did I mentions Documentation? – yes, I do internal audits for Transcat, and I am a firm believer, as is the FDA, that if it isn’t written, it didn’t happen
- I didn’t have another supportind documentation to justify a shorter time interval Why Jan 15th and not January 21st (The calibration date?) Because the unit was received by the Lab on the 15th which means that it could not have been used by the customer after that date. CLICK Document removal from Service
- Where was it used? This is probably the biggest challenges start, it is the weakest link in the Chain of Traceability and where traceabity most likely breaks down. Especially in large facilities using hand held and bench level equipment A robust instrument control program that track the location of every calibrated unit is criticle. Next identify where this instrument has been used during the questionable period. The ease of identifying potential at risk product depends upon the design of the end users processes and systems. Test equipment that moves around without tracking its location is going to be the biggest problem. A robustly designed system with strict instrument control procedures will be able to identify exactly where any given instrument was located for any given time frame Here at Transcat …. CLICK for Tips I am also a 6 Sigma Green Belt and a Lean Enterprise Practitioner
- GmetP Tip use the full identification fields Transccats system consist of 3 customer usable fileds
- Here is why I recommend the sequence What, When Where Determine exactly what measurements were being made at a given location, during the time frame in question. This information will likely be found in the end users procedures, work instructions, or an engineering specification. The objective at this step is to determine whether the out-of-tolerance instrument could have affected any of the products. We are not determining the impact yet, we are not making a decision here, just collecting evidence. and all revisions that were in effect during the time frame in question. Were any of the out-of-tolerance functions, parameters, ranges and test points used to make the measurements listed in the process documentation? If the answer is no, congratulations, your evaluation has ruled out the potential risk to product. Now you just have to completely document the steps you have taken, your conclusion and justification, as any auditor or FDA Inspector will tell you, if it isn’t written, it didn’t happen, you must provide objective evidence.
- How was it used? Determine exactly what measurements were being made at a given location, during the time frame in question. This information will likely be found in the end users procedures, work instructions, or an engineering specification. The objective at this step is to determine whether the out-of-tolerance instrument could have affected any of the products. This can be accomplished by reviewing the process documentation, and all revisions that were in effect during the time frame in question. Were any of the out-of-tolerance functions, parameters, ranges and test points used to make the measurements listed in the process documentation? If the answer is no, congratulations, your evaluation has ruled out the potential risk to product. Now you just have to completely document the steps you have taken, your conclusion and justification, as any auditor or FDA Inspector will tell you, if it isn’t written, it didn’t happen, you must provide objective evidence. Analyzing the impact If the answer is yes then the real work begins. You have to quantify the severity of the impacted products or services. In order to effectively complete this analysis, a thorough understanding of the affected process is necessary and a working understanding of tolerances and the application of uncertainties is extremely helpful. Every bit of measurement information at your disposal allows you to make additional distinctions, observations, calculations and improves the quality and confidence in your conclusions and recommendations for further actions, don’t forget to document them!
- To determine if the lab standard’s OOT condition impacted customer measurements, the magnitude and direction of the unexpected error must be determined. The unexpected error is the portion of the As-Found measurement outside of the accuracy limits, either above the upper tolerance limit or below the lower tolerance limit. This is the only amount of error we must take into account when evaluating the reported measurements because the lab standards accuracy has already been accounted for in the lab uncertainty budget. There are two questions to be answered, what is the error of the lab standard, and did this error cause a change in the As-Found compliance statement of In-Tolerance or Out-of-Tolerance.
- Expected –We though the bath was at 100.00°C ± 0.06°C , but our UUT was “lying” to us Actual – Here where the questions will fly Corrected – We only correct for the Unexpected error. Why not correct for the entire 0.10°C Error? Remember, we selected test equipment with tolerances that had an acceptable limits that we could live with What we can’t live with is the additional unexpected error.
- A measurement assurance program is more than a calibration program; it is a thought process, a link that relates measurements through the entire produce life cycle, from concept to end product. Hopefully this approach and general guidelines will ease the burden to solving one of the most dreaded situations in the measurement world: the evaluation of an out-of-tolerance instrument and its potential impact.
- The cost of a single product recall will far exceed any cost associated and time spent in the four steps (what, when, where, and how) in the investigation process which eliminates additional products to be analyzed is well worth the time. Remember the objective is to filter out as many possible items that do not need closer analysis so you can get to the ones where detailed analysis is required in order to quantify the impact to the products or services provided. All this evaluation and analysis is a tremendous amount of work. A well thought out electronic system linking instrumentation to processes and product traceability as part of a measurement assurance program can ease the burden of out-of-tolerance evaluations and analysis.