Primer on the application of statistical significance testing for business research purposes. 1) How to use statistics to make more informed decisions (and when not to use). 2) Highlight differences between statistics in science vs business. 3) Highlight assumptions, limitations and best practices.

1. 1
STATISTICAL
SIGNIFICANCE
&
BUSINESS RELEVANCE
How to apply statistics for business decision-making.

2. Is Superman stronger than Charlie Brown?
2
Some tests are easy to analyze
Don’t need statistical testing

3. Is Superman stronger than Batman?
3
Data does not show a clear
overwhelming winner
Use statistical significance to
determine if their findings are valid.

4. Use statistics only if valuable.
• If statistics can benefit me, I will use it.
• If statistics poses a threat, then I will not.
4

5. Lets take an example…
5
Research Question
Determine the
effectiveness of
share-a-coke campaigns.
Business Question
Coke needs to select an
ad to go to market with
a $10M investment.

6. Research needs 3 Things…
1. Who are Coke Users/Non-Users?
– Sample to recruit to include in study
2. What will we compare the adcepts to?
– Independent Variable
3. What is a valid & reliable measure of effectiveness?
– Outcome Measure / Dependent Variable
6

7. 1. SAMPLING: Who are Coke Users/Non-Users?
7
populationconclusions based
on the sample
sample
generalization to the
population
hypotheses

8. SAMPING BIAS: The greater the variation in the
underlying population, the larger the sampling error.
8 Gallo, 2016; HBR

9. 2. INDEPENDENT VARIABLE: What will we compare
the adcepts to determine the effectiveness?
9
CONTROL
No-Adcept Control
Previous Best
Competitor

10. Compare Superman to: Result Interpretation
Superman is
significantly stronger;
Lets choose him.
Superman is not
significantly stronger;
(no statistical differences);
Lets compare the
differences in strength
We may keep looking
The comparison is KEY for interpreting data in
order to make relevant business decisions
10
Business Question: Should we choose Superman to help us win?

11. Compare to:
Hypothetical
Result:
Confidence in
Making Decision
Will it perform better
than nothing?
Significant Not as confident
Wil perform better
than our previous
best?
Significant More confidence
Wil perform better
than our competition?
Significant More confidence
11
Business Question: What adcept will be most effective campaign?
CONTROL
The comparison is KEY for interpreting data in
order to make relevant business decisions

12. Choosing Comparisons Take Aways…
1. Choose comparisons that are grounded in the business decision
context.
2. Instead of thinking in terms of statistical significance, p values
(.05), and confidence intervals (which are limited for business
application) think in terms of methods to increase
subjective confidence to make the decision (better
comparisons, additional comparisons, better dependent
variables, using effect sizes).
12

13. The Two Hypothesis!
Null Hypothesis Treatment
There is NO difference between
the two groups
There is a difference between the
two groups
= no effect = there is an effect
13
Sig Differences?

14. When you perform a test of statistical significance you
usually reject or do not reject the Null Hypothesis (H0).
The null hypothesis
– no difference between treatment
effects
or
– no association between variables
14

15. Significance Testing
Statistically significant mean
difference at p < .05 tells us that if
we sampled many pairs of groups
from the same hypothetical
population, we would expect to get
a difference as large as the
observed result or larger with no
more than 5% of the groups as the
result of sampling error, given that
the null hypothesis
15

16. SO WHAT?
16

17. Statistical Significance
• P values, or significance levels, measure the strength
of the evidence against the null hypothesis
Significance can only tell the likelihood
that a relationship exists
It can’t tell whether or not it’s important.
17 Sterne, 2001

18. Much value in making business decisions is with effect sizes. That is,
how much stronger is Superman than Batman?
18
SIGNIFIGANCE
A P value describes the likelihood of
a true relationship between
X (Superman) and Y (Batman)
MAGNITUDE & EFFECT SIZE
Effect size show the magnitude or
size of the relationship between
X (Superman) and Y (Batman)?

19. What is statistical significance…
• A result has statistical significance when it is very
unlikely to have occurred given the null hypothesis.
• More precisely, the significance level defined for a
study, α, is the probability of the study rejecting the
null hypothesis, given that it were true
19

20. Effect Size is a more practical for business purposes.
20
CONTROL
EFFECT SIZE
How much more effective?
SIGNIFIGANCE
Is there a relationship?
?

21. What are the limitations of
inferential statistics?
21

22. People overvalue the role of statistics
Statistical significance testing is often misused:
1. Endow them with capabilities they do not have.
2. Utilize them as the sole approach to analyzing data.
We should…
1. Become aware of the limitations of most inferential statistics.
2. Augmenting statistics with other information and research
approaches.
22 Sawyer & Peter, 1983

23. What is important to a Manager
may not be statistically significant.
Alternatively, what is not important to a Manger
may be statistically significant.
23
People overvalue the role of statistics

24. Successful use of statistics is different…
24
Academics Business
Prove a point Make a decision
Statistical Significance &
High Confidence intervals
Practical Significance &
Subjective confidence (Decision certainty)

25. Statistical Tests are
NOT completely
objective.
The statistical
significance level
obtained is strongly
influenced by
subjective decisions by
the researcher.
25

26. 26
1. 1 or 2-tailed test
2. Level of significance
3. Number of observations
1. Standard deviation
2. Amount of deviation from
the null hypothesis
Controlled
by researcher
Influenced
by researcher
Sawyer & Peter, 1983
Statistical Tests are NOT Completely Objective.

27. 4 Misinterpretations of Significant Results
1. Probability of the Null Hypothesis
• Probability that the results occurred because of chance
2. Probability of the Results being Replicated
• Probability that results will be replicated in the future
3. Probability of Results Being Valid
• Probability that the alternative hypothesis is true
4. Sample Size and Probability of the Research Hypothesis
• Confusion about the sample size and level of statistical significance
27 Sawyer & Peter, 1983

28. 5 Ways to turn “Non-significant” into “Significant”
1. Increasing the sample size
2. Increasing the reliability of the measures
3. Changing post-hoc the acceptable level of statistical significance
4. Changing from 2-tailed to 1 tailed test
5. Obtaining better control over non-manipulated variables
28 Sawyer & Peter, 1983

29. 3 Common Misunderstandings with sample size and
statistical significance
1. Relationship between sample size and level of statistical significance
implied that more confidence should accompany the result of the study
had a large sample size rather than a small one.
2. Larger samples do reduce likely sampling error because their estimates
more closely approximate the population parameters, but it should also
be clear that differences in the amount of sampling error are included
explicitly in the computation of statistical significance tests.
3. There should NOT be a bias against statistically significant results
obtained from properly selected small samples.
29 Sawyer & Peter, 1983

30. Statistics Humor…
30

31. Critics of statistics…
• "Reliance on merely refuting the null hypothesis...is basically unsound,
poor scientific strategy, and one of the worst things that ever happened in
the history of psychology.”
• "Can you articulate even one legitimate contribution that significance
testing has made (or makes) to the research enterprise (i.e., any way in
which it contributes to the development of cumulative scientific
knowledge)?”
• Is there any study wherein statistical significance improves decision-
making?
31 Schmidt, 1996; Meehl, 1978

32. Citations
1. Gallo, A. (2016). A Refresher on Statistical Significance. Harvard business
review.
2. Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir
Ronald, and the slow progress of soft psychology. Journal of consulting
and clinical Psychology
3. Sawyer, A. G., & Peter, J. P. (1983). The significance of statistical
significance tests in marketing research. Journal of marketing research.
4. Schmidt, F. L. (1996). Statistical significance testing and cumulative
knowledge in psychology: Implications for training of researchers.
5. Sterne, J. A., & Smith, G. D. (2001). Sifting the evidence—what's wrong
with significance tests?. Physical Therapy.
32

33. Jason Martuscello
33
jason@prospectionsciences.com