An extension on a series about hypothesis testing, this lesson introduces some statistical concepts that are fundamental to most hypothesis testing.

1. Section & Lesson #:
Pre-Requisite Lessons:
Complex Tools + Clear Teaching = Powerful Results
Hypothesis Testing: Statistical Laws
and Confidence Intervals
Six Sigma-Analyze – Lesson 11
An extension on a series about hypothesis testing, this lesson introduces
some statistical concepts that are fundamental to most hypothesis testing.
Six Sigma-Analyze #10 – Hypothesis Testing – Formal and
Informal Sub-Processes
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted by any means
(electronic, mechanical, photographic, photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.

2. Some Statistical Laws
o There are 3 general laws commonly associated with statistics:
• The Law of Averages, the Law of Large Numbers (LLN), and the Central Limit Theorem (CLT).
o Law of Averages
• This is actually not a law but a lay term generally applied by people unfamiliar with statistics.
• It is a belief that the outcome of random events will eventually “even out”.
 Example 1: Belief an event is “due” to happen.
– If you flip a coin 10 times and get heads all 10 times, this belief presumes that you’re “due”
to get tails next. However, the probability of getting tails doesn’t change; it’s still 50/50.
– However, since getting heads 10 straight times is unexpected, then it’s more likely that the
coin is unevenly weighted (bias). If so, there’s a greater probability of getting heads again.
 Example 2: Belief a sample’s average must equal its expected value.
– If you flip a coin 100 times, there’s only an 8% chance there will be exactly 50 heads.
 Example 3: Belief that a rare occurrence will happen given enough time.
– "If I send my résumé to enough places, someone will eventually hire me."
– This may actually be true assuming nonzero probabilities and that the number of trials is
really large enough; the law of averages is then simply the Law of Large Numbers (LLN).
o Law of Large Numbers (LLN)
• This is a legitimate theorem that the average of the results obtained from a large number of
samples will be close to the expected value and will get closer as you obtain more samples.
 Example: The expected average value of a six-sided die is (1 + 2 + 3 + 4 + 5 + 6) ÷ 6 = 3.5.
– Roll the die 10 times, the average of those 10 rolls will be close to 3.5.
– The more trials or sets of 10 rolls you record, then the closer the cumulative average will be to 3.5.
• This law is the premise for the Central Limit Theorem (CLT) applied in statistical testing.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
2

3. Central Limit Theorem (CLT)
o The statistical tests used in hypothesis testing are founded on the CLT.
• Understanding this theorem can help you understand the basis for most statistical tests.
o What is the Central Limit Theorem (CLT)?
• Technically, the means of random samples from any distribution (normal or non-normal) with a
mean of μ and a variance of σ2 will have the following:
 An approximately normal distribution.
 A mean equal to μ.
 A variance equal to σ2/n.
o Example: Rolling Dice
• Suppose you roll a pair of dice 10 times and write down the average of all the rolls. If you
continue rolling the dice and writing down the average value after each set of 10 rolls, then:
 The averages of each of those sets of rolls will be about the same.
 The more you roll, the variance between the values will become more & more narrow (less variation).
• What if the dice are weighted (bias)?
 The theorem results are the same, except the average will shift.
o What can we conclude from the CLT?
• The more samples you collect, the more confident
you can be about what the mean is for all
samples of the population.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
3
N = 5 sets of dice rolls
N = 10 sets of dice rolls
N = 20 sets of dice rolls

4. Confidence Intervals (CI)
o What is a confidence interval (CI)?
• It represents the range (lower & upper bounds) in which the population mean should reside
based on the data in the sample.
• Let’s look at the impact of the CI using the prior dice example:
• The spread of the range is dependent on the size of the sample (more samples will reduce the
interval) and the desired confidence (95% confidence will be more narrow than 99%).
 Remember the Standard Error of the Mean (SE Mean)? It is the descriptive statistic used to calculate the
confidence interval. To get 95% CI, calculate SE Mean by 2, or by 3 to get 99%.
• Remember, statistics are intended to help you make inferences about a population.
 A mean of 57 with a 95% confidence interval of 55 and 59 implies that although the sample mean is only
57, you can be 95% confident that the population mean will fall somewhere between 55 and 59.
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
4
Based on 5 sets of dice rolls, the population
mean should fall within this interval.
Based on 10 sets of dice rolls, the population
man should fall within this more narrow interval.
Based on 20 sets of dice rolls, the population
man should fall within this more narrow interval.

5. Using CIs in Statistical Tests
o How do statistical tests use the confidence intervals (CI)?
• Statistical tests generally compare the different CIs between factors to see if a difference exists.
o Example: Compare efficiency between two systems
• Scenario: 50 similar transactions were each run through two different systems. System A’s time
was 180 seconds and System B’s time was 160 seconds (assume standard deviation of about 1/3).
 Is System B statistically faster? What is the best/worst case difference in efficiency for System B?
o Bottom Line: Increase your sample size to decrease (narrow) your confidence interval.
5
Employees
Contractors
0 5 10 15 20
Large overlap indicates the population
mean may be the same for both
groups; therefore there is no difference
between them (high P-value).
N = 50
Employees
Contractors
0 5 10 15 20
Small overlap indicates a smaller chance
the population mean may be the same for
both groups; therefore there may be no
difference between them (small P-value).
N = 100
Employees
Contractors
0 5 10 15 20
No overlap indicates the population
mean is different for both groups;
therefore there is a difference between
them (low or no P-value).
N = 200
200195190185180175170165160155150145
System B
This overlap suggests both may have the same speed.
System A
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.

6. Practical Application
o Scenario:
• You have two sets of continuous data that you’re comparing; each set has 100 values. The first
dataset has an average value of 127 with a confidence interval of 117 to 137, and the second
dataset has an average value of 132 with a confidence interval of 123 and 140.
o Try to answer the following questions based on the above scenario:
• Would you consider the confidence intervals to be wide or narrow?
• Would you expect the P-value to be high, medium, or low?
• Is there a statistical difference between these two datasets?
 How confident (roughly) would you be in your conclusion?
• How would the analysis change if you were only able to collect 50 samples per dataset?
• How would the analysis change if you collect 300 samples per dataset?
Copyright © 2011-2019 by Matthew J. Hansen. All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system, or transmitted by any means (electronic, mechanical, photographic,
photocopying, recording or otherwise) without prior permission in writing by the author and/or publisher.
6