The first of an extended series on conducting a measurement system analysis (MSA) to help test the reliability of collected data.

1. Section & Lesson #:
Pre-Requisite Lessons:
Complex Tools + Clear Teaching = Powerful Results
MSA - Overview
Six Sigma-Measure – Lesson 28
The first of an extended series on conducting a measurement system
analysis (MSA) to help test the reliability of collected data.
Six Sigma-Overview #2 – Risk Analysis – The Reason We Use
Statistics
Six Sigma-Measure #2 – The Necessity of the Measure Phase
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted by any means
(electronic, mechanical, photographic, photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.

2. Measurement System Defined
o What is a measurement system?
• It refers to the method used for collecting data or calculating metrics.
• Some examples of measurement systems include the following:
o How do you know you can trust the measurement system?
• Many metrics are taken for granted; but how do you know they’re accurate?
• What is the risk of having errors in the measurement system?
• What if no measurement system exists for what you need to measure? How do you create it?
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
2
Average Call Duration
Total Call Duration / Call Volume
OIBDA
Operating Income + Depreciation +
Amortization + Tax + Interest
(“OIBDA” stands for “Operating Income
Before Depreciation & Amortization”.)
Churn
# of customer cancellations /
Total # of subscribed customers
Average Revenue per Unit (ARPU)
Total Revenue / Total # of Units
(A unit can be a customer, product, etc.)

3. MSA Defined
o What is a measurement system analysis (MSA)?
• It is a method for testing and validating the accuracy & precision of a measurement system.
• It measures the level of risk in the reliability of a measurement system.
• It’s not just calibration; it’s a more formal and methodical analysis of the measurement system.
o What’s the difference between Accuracy & Precision?
• Accuracy measures how correct a measurement is to a known standard or “master”.
• Precision measures variation within the same and between different operators.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
3
Accurate but not Precise Precise but not Accurate Accurate and Precise

4. Measuring Accuracy and Precision
o How do we measure accuracy & precision in a MSA?
• Accuracy and precision are measured from 3 perspectives:
o Some MSA tools can also measure by additional perspectives:
• Linearity – is there bias over the operating range?
• Stability – is there variation over time in the data process or measurement system?
• Resolution/Discrimination – is the measure granular enough to detect small bias or variation?
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
4
Accuracy
Is the data recorded
correctly (i.e., is there
bias in the data)?
Repeatability
Is the data recorded
consistently by each
(within) operator?
Reproducibility
Is the data recorded
consistently between
different operators?
Accuracy Precision

5. General Flow of the MSA
o Below is an overview of how a MSA is conducted and the 3 perspectives analyzed.
• MSAs generally involve 3+ operators doing 3+ review trials of the same items.
• This represents a blind study where the operators don’t know they’re processing the same info.
• This example is what is commonly used in a transactional environment.
 Transactions to review may include customer calls, invoices, purchases, etc.
 In a manufacturing environment, it would be a part or product that is reviewed by each operator.
5
 = 1st review trial
 = 2nd review trial
 = 3rd review trial


















Master
Accuracy:
Compare each
output to Master
Repeatability:
Compare outputs
within same operator
Reproducibility:
Compare outputs
between operators
Items to
Review
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.

6. Is the MSA Necessary?
o A Measurement System Analysis (MSA) can get very complex and take extra time.
• The more complex (a lot of variables) and more operator-dependent (manual processes &
measurements) in the measurement system, then the more time is required to do the MSA.
o If it takes so much time, then is a MSA really necessary?
• Maybe. Like everything else, it depends on RISK. That is, what is the risk of not doing a MSA?
o How do I measure the risk of not doing a MSA?
• You can’t; there isn’t a formal, objective way to measure the risk of not doing a MSA.
• Even so, you can subjectively assess the risk by asking yourself the following:
 Are you at least 90% confident you can trust the accuracy of the measurement system?
 Are you at least 90% confident you can trust the precision (lack of variation) of the measurement system?
 Is the measurement system not based on human involvement (e.g., time & motion, tick sheets, etc.)?
 Is the speed of finishing the project more important than the accuracy or quality of the analysis?
– This is a balance of comparing risk vs. reward.
 If you later discover the data was unreliable, would it be acceptable to redo the data collection & MSA?
• If the team answers “Yes” to these questions, then it may be acceptable to skip the MSA.
 This should be reviewed with the Champion & Sponsor so they’re aware of the risks of skipping the MSA.
o Since MSAs can take on many forms and get very complex; let’s keep it simple.
• This module introduces the concepts and use of simple MSA tools that are most common.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
6

7. Overview of Building the MSA
o There are 4 general steps for building the MSA:
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
7
1. Plan
the MSA
2. Conduct
the MSA
3. Analyze
the Results
4. Improve
measurement
system
This involves determining:
• What needs to be measured
• What MSA tools will be used
• Who will be part of the study
• What samples/master to use
• How to conduct the study
This involves:
• Scheduling the study
• Leading the study
• Collecting data from the study
This involves:
• Compiling all collected data
• Processing data in MSA tools
• Interpreting analysis results
This involves determining:
• If MS needs improvement
• How to improve the MS
• Redoing MSA (as needed)

8. Practical Application
o Identify at least 3 completed projects or initiatives you worked in your organization.
• At any point was there any data or critical information that was accused of being unreliable?
 If so, what caused the accusation to be raised?
 Was the accusation true (i.e., was it true that the data was unreliable)?
– If so, why wasn’t the reliability of the data tested in advance?
– How could a MSA have helped catch or prevent the unreliability of the data?
– Otherwise if the data was reliable after all, then what could’ve been done differently to prevent the accusation?
 If there was no accusation at all, then why not (i.e., what kept anyone from challenging the data)?
– Should the data have been challenged, but perhaps no one was willing nor knowledgeable enough to challenge it?
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
8