Practical Tools for Measurement Systems Analysis presented at the American Statistical Association's Orange County and Long Beach Chapter quarterly meeting

1. Practical Tools for Measurement Systems Analysis
Created and presented by: Gabor Szabo
American Statistical Association, Orange County and Long Beach Chapter Meeting
December 11, 2019

2. Practical Tools for Measurement Systems Analysis
Presentation Outline
• Overview of use of applied statistics at a medical device manufacturer
and specifically in quality engineering
• Measurement Systems Analysis overview
• Practical approach to Measurement Systems Analysis
• Tools and Approaches to augment or replace conventional methods
• Examples

3. Practical Tools for Measurement Systems Analysis
Research &
Development
Manufacturing Post Market
• Validation
• Equipment
• Process
• Product
• Risk Assessment
• Software Quality
• Equipment
• Product
• Systems
• Metrology
• Calibration
• Equipment
Evaluation
• Quality Inspection
• Process QE Support
• Compliance
• External Audit
• US FDA
• International
• Internal Audit
• Supplier Quality
• Qualification
• Audit
• Document Control
• Product Complaint
• Investigation
• Analysis/ Reporting
• Field Corrective Actions
• Product Recall
• Correction
• Advisory Letter
•Other Quality Related Functions: Business Excellence (Six Sigma, Lean), Regulatory, Clinical, Biostatistics
Uses of applied statistics in a medical device company

4. Practical Tools for Measurement Systems Analysis
• Acceptance sampling
• Measurement Systems Analysis
• Capability studies (Cp, Cpk, Pp, Ppk)
• Tolerance Intervals
• Superiority/Non-Inferiority/Equivalence tests
• Process Control Charting (SPC)
• Design of Experiments
• Regression
Applied Statistics in Quality

5. Practical Tools for Measurement Systems Analysis
Intended Uses of a Measurement

6. Practical Tools for Measurement Systems Analysis
Measurement Systems Analysis
• Activities to identify, quantify and unconfound measurement error
from the variation of the product/objects being measured

7. Practical Tools for Measurement Systems Analysis
System
• Gage
• Fixtures
• Appraisers
• Preparation methods
• Measurement procedure
• Environmental conditions
• Objects being measured

8. Practical Tools for Measurement Systems Analysis
Terms
• Repeatability: same objects, gage, appraiser measured multiple times
• Stability: same objects, appraiser and gage measured over time
• Reproducibility: Difference between appraisers
• Method Bias: Difference between different types of gages or multiple
gages of the same type in different locations

9. Practical Tools for Measurement Systems Analysis
Terms – continued
• Bias: The difference between the true value and the measured value
• Linearity: The consistency of bias over the range of values

10. Practical Tools for Measurement Systems Analysis
Conventional MSA Approach
• Based on AIAG MSA Reference Manual from the automotive industry
• Repeatability and Reproducibility – Gage R&R Studies
• Sample size: 5-10 product samples (parts), measured 2-3 times (trials)
by 2-3 appraisers
• %Tolerance and %Study Variation metrics
• Acceptance criteria: 10/20/30 % rule

11. Practical Tools for Measurement Systems Analysis
Repeatability
• The statistical model for repeatability error (or random measurement
error) is the normal distribution (Laplace, 1810)
• Repeatability error calculated as a standard deviation (σ 𝑟𝑒𝑝𝑒𝑎𝑡𝑎𝑏𝑖𝑙𝑖𝑡𝑦) in
the ANOVA Gage R&R method, and in most methods
• Metrics used to characterize repeatability error:
o Probable Error (Wheeler): 0.675 ∗ σ 𝑟𝑒𝑝𝑒𝑎𝑡𝑎𝑏𝑖𝑙𝑖𝑡𝑦 (or R bar from R Chart)
o %Tolerance:
6 ∗ σ𝑟𝑒𝑝𝑒𝑎𝑡𝑎𝑏𝑖𝑙𝑖𝑡𝑦
𝑇𝑜𝑙𝑒𝑟𝑎𝑛𝑐𝑒
* 100 (per AIAG Method)
o %Study Variation:
σ𝑟𝑒𝑝𝑒𝑎𝑡𝑎𝑏𝑖𝑙𝑖𝑡𝑦
σ 𝑝𝑟𝑜𝑑𝑢𝑐𝑡
* 100 (per AIAG Method)

12. Practical Tools for Measurement Systems Analysis
Repeatability
± 3 * σrepeatability –> %Tolerance
σrepeatability
• %Tolerance: 99.73% confidence
interval around mean of
measurements over tolerance
• Probable Error: 50% confidence
interval around mean of
measurements
USLLSL
± 0.675 * σrepeatability –> Probable Error (or R bar)
50% CI
99.73% CI

13. Practical Tools for Measurement Systems Analysis
Reproducibility
• Calculated as a standard deviation in the Range (AIAG) and ANOVA
Gage R&R methods
• Operator differences are almost always a fixed effect (bias), not a
random effect
• Standard deviations can only be calculated for random effects
(variations)

14. Practical Tools for Measurement Systems Analysis
Reproducibility
Operator bias
6 * σreproducibility
Reproducibility error
Op 1
Op 2 Op 3
σreproducibility
• Operators are not randomly
distributed, as the calculation
method suggests
• Reproducibility error is
overstated by the conventional
MSA statistic

15. Practical Tools for Measurement Systems Analysis
Reproducibility
• ANOM (Analysis of Means)
• To assess whether the differences
between the operator means are
statistically significant
• Practical significance should also be
assessed

16. Practical Tools for Measurement Systems Analysis
“PGA” Approach
• Practical
• Graphical
• Analytical

17. Practical Tools for Measurement Systems Analysis
Practical
• Going into the study: what is the purpose of the measurement?
• How big of a difference am I trying to detect? “Signal to Noise”
• What are my specification limits? Where is my process running?
• Is there anything about the raw data that visually stands out?

18. Practical Tools for Measurement Systems Analysis
Graphical
• Graphical look
• Is there anything about the data that graphically stands out?
• Chart study results and look for clues – shifts, drifts, patterns,
special causes etc.

19. Practical Tools for Measurement Systems Analysis
Analytical
• This should be the last step of your analysis
• Be careful about what you choose to be the metric of the
analysis
(“score mentality”, see ASA’s statement on p-values from 2016)

20. Practical Tools for Measurement Systems Analysis
1 2Operator
Part
Trial 1 2 3 4
Tree Diagram
Example of PGA Approach: Gage R&R for optical wall thickness measurement
1 2 3 1 2 3
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
• Optical
measurement
• Extruded
components
• Specification:
.012”-.013”

21. Practical Tools for Measurement Systems Analysis
Practical
Visual – Raw data
Sample Operator Trial WT
1 1 1 .0120
2 1 1 .0124
3 1 1 .0120
1 1 2 .0120
2 1 2 .0124
3 1 2 .0119
1 1 3 .0120
2 1 3 .0124
3 1 3 .0119
1 1 4 .0120
2 1 4 .0124
3 1 4 .0120
1 2 1 .0120
2 2 1 .0125
3 2 1 .0119
1 2 2 .0119
2 2 2 .0125
3 2 2 .0118
1 2 3 .0119
2 2 3 .0125
3 2 3 .0119
1 2 4 .0120
2 2 4 .0124
3 2 4 .0118
Specification: .012”-.013”

22. Practical Tools for Measurement Systems Analysis
Graphical
Tool: Multi-Vari Chart
• Ability to compare variation
across factors
• Bottom-up
• Ability to look for patterns in
the data
Operator
Part
Trial
321
0.0125
0.0124
0.0123
0.0122
0.0121
0.0120
0.0119
0.0118
321
1
Sample
WT
2
1
2
3
4
Trial
Multi-Vari Chart for WT by Trial - Operator
Panel variable: Operator
Are these
special
causes?

23. Practical Tools for Measurement Systems Analysis
Graphical & Analytical I.
Tool: Xbar-R behavior chart
• Charts staged by operator
No special causes
present
Repeatability
Part-to-Part plus
Operator Effects
• Xbar chart shows repeatability vs.
part-to-part variation

24. Practical Tools for Measurement Systems Analysis
Graphical & Analytical II.
Tool: Xbar-R behavior chart
• R bar: represents average
repeatability error
• Ranges above control limit on R
chart need to be investigated
Graphical – Multi-Vari Chart

25. Practical Tools for Measurement Systems Analysis
Comparison: Multi-Vari and Xbar-R behavior charts
321
0.0125
0.0124
0.0123
0.0122
0.0121
0.0120
0.0119
0.0118
321
1
Sample
WT
2
1
2
3
4
Trial
Multi-Vari Chart for WT by Trial - Operator
Panel variable: Operator
• Trends, patterns
• Difference between appraiser averages
• Appraiser effects, appraiser-part interactions
• Quantifies repeatability error (R bar and UCL)
• Identifies special causes
• Appraiser effects, appraiser-part interactions
Graphical Graphical, Analytical

26. Practical Tools for Measurement Systems Analysis
Examples
#1
Is there anything about the
charted data that stands out?
Next slide

27. Practical Tools for Measurement Systems Analysis
Examples
#1
• R chart: none of the ranges
appear to be above the UCL
• Repeatability error exceeds part-
to-part variation
Next steps:
• Investigate and eliminate special
causes
• Assess whether further
reduction of repeatability error
is necessary

28. Practical Tools for Measurement Systems Analysis
Examples
#2
Is there anything about the
charted data that stands
out?

29. Practical Tools for Measurement Systems Analysis
Examples
#2
• Resolution issue
• One of the operators (TN)
rounded/truncated their
measurement results
Next steps:
• Re-do study without rounding
measurements
• Re-evaluate results

30. Practical Tools for Measurement Systems Analysis
Examples
#3
Is there anything about the raw data
that stands out?
Sample Trial Operator Dimension
1 1 1 .04121
1 2 1 .04119
1 3 1 .04115
1 4 1 .04112
2 1 1 .04138
2 2 1 .04134
2 3 1 .04128
2 4 1 .04127
3 1 1 .04114
3 2 1 .04112
3 3 1 .04109
3 4 1 .04108
1 1 2 .04120
1 2 2 .04116
1 3 2 .04114
1 4 2 .04110
2 1 2 .04113
2 2 2 .04111
2 3 2 .04102
2 4 2 .04101
3 1 2 .04138
3 2 2 .04137
3 3 2 .04135
3 4 2 .04131

31. Practical Tools for Measurement Systems Analysis
Examples
#3
Is there anything about the charted
data that stands out?
• R chart: none of the ranges
appear to be above the UCL
• Repeatability error does not
exceed part-to-part variation

32. Practical Tools for Measurement Systems Analysis
Examples
#3
Is there anything about the
charted data that stands out?
• Decreasing pattern trial-to-
trial
• Appraiser-part interaction
Next steps:
• Investigate the physics of
the measurement or
manufacturing process
• Investigate potential root
causes

33. Practical Tools for Measurement Systems Analysis
Youden Plot – An Alternative Approach
• Other name: Measurement Discrimination Plot
(Dr. Wheeler: EMP III)
• A square scatter diagram with a 1:1 45-degree line
(line of perfect agreement)
• Two measurements are captured to graphically
assess repeatability, reproducibility or bias

34. Practical Tools for Measurement Systems Analysis
• Measurement 1 and 2 are taken and
captured
• Measurement 1 on the X axis
• Measurement 2 on the Y axis
• If there is agreement between Measurement
1 and Measurement 2, the data point falls on
the 45-degree line
• Specification limits (if any) are plotted on
both axes
Measurement 1
Measurement2
Youden Plot – An Alternative Approach

35. Practical Tools for Measurement Systems Analysis
• Visualization of repeatability error against
product variation and specification limits
(Graphical)
• To be used in conjunction with R chart
(Graphical and Analytical)
• Sample size of 30 allows for a good estimate
of product variation
Youden Plot – An Alternative Approach

36. Practical Tools for Measurement Systems Analysis
Comparison: Youden Plot – Multi-Vari Chart
• Ideal when number of trials is > 2
• Ideal for graphically analyzing 5-10 parts
• Ideal for analyzing patterns
• No visual comparison of repeatability error to specification
• Number of trials is 2 (X and Y axes)
• Ideal for graphically analyzing 20 or more parts
• Visual comparison of repeatability error to specification
and product variation

37. Practical Tools for Measurement Systems Analysis
• After assessment of statistical significance
through ANOM
• Assessment of practical significance
Youden Plot, Reproducibility Assessment
Difference between
operator means

38. Gage/ Method 1
Bias
Gage/Method2
 Graphical tool to assess method bias
 Measurement from Gage/Method 1 on X axis
 Measurement from Gage/Method 2 on Y axis
 If there is perfect agreement between the two
methods, the data point falls on the 45-degree
line
 Specification limits (if any) may be plotted on
both axes
Practical Tools for Measurement Systems Analysis
Youden Plot for Method Comparison

39.  Graphical and Analytical tool to
assess method bias
 Averages of Gage/Method 1 and 2
measurements on X axis
 Differences [Gage/Method 2-
Gage/Method 1] on Y axis
 Centerline: mean difference
 ULA and LLA: mean difference +/- 2
standard deviations
DifferencebetweenMethod1and2
Average of Method 1 and 2
Practical Tools for Measurement Systems Analysis
Bland-Altman Plot for Method
Comparison

40. Method Comparison – Youden and Bland-Altman
Plots
1:1 line on Youden plot
Mean line on Bland-Altman plot
Comparison: Youden and Bland-Altman Plots
• Individual results from method 1 vs. method 2
• Visual assessment of bias and expected range
of differences
• Average of results from two methods vs. individual
differences
• Quantifies bias and range of expected individual differences
Graphical Graphical, Analytical

41. Example 1
Comparison of two gages
• Characteristic of interest: flow rate
• Uson and Z axis air flow testers
• Two groups of parts spanning range of ~ 3.5 units (sccm)
• n = 30 (2 x 15)
Observations?
• Data points are relatively close to 1:1 line on Youden plot
• Mean difference is 0.03 units
• Lower group mean difference ~ 0.1 units
• Higher group mean difference ~ -0.05 units

42. Simulation
Flowtest,assylevel
Example 2
Comparison of two methods
• Five groups of parts spanning a range of ~ 7 units (sccm)
• n = 100 (5 x 20)
Observations?
• Varying degrees of bias across groups of parts – non-
linearity
• Mean difference is -1.33 units

43. Example 2 – cont.
Comparison of two methods
Observations?
• Difference in data spread between simulation
method and original method
Flowtest,assylevel
Simulation
Difference in repeatability error

44. Practical Tools for Measurement Systems Analysis
Takeaways
• Use the Practical-Graphical-Analytical (PGA) approach to assess
your measurement systems analysis results, look for patterns and
take action based on findings
• The Youden Plot is a great tool to assess measurement error in a
practical way
• I plan on releasing online content on the subject early next year. If
you are interested, leave your contact info on the evaluation form