A funny presentation on the power of a statistical test of hypothesis.

1. I’VE GOT THE
POWER!!
F R A N C O I S – T E C H N I C A L T E A M M E E T I N G

2. ONE EVENING …

3. ARE YOU HAPPY TO GAMBLE?
•She gives me a pair of dice from the left
pocket of her jacket
•She takes another pair of dice from the
right pocket of her jacket …

4. I THROW MY DICE …

5. SHE THROWS HER DICE AND SHE
SMILES AT ME …

6. QUESTION: HOW SHOULD I REACT??
OR

7. WHAT ARE THE HYPOTHESES?
• H0: The dice are fair
• H1: The dice are loaded (i.e. the probability to get 6 is greater
than the probability to get 1 – 5, equal probability each i.e.
p(1) = p(2) = … = p(5)).

8. HYPOTHESIS TESTING: REMINDER
H0 is true H1 is true
Accept H0 1 - α β
Reject H0 α 1 – β (power)
Conditional probabilities:
α = p(reject H0 | H0)
β = p(do not accept H1 i.e. accept H0 | H1)
1 - β = p(accept H1|H1)

9. POWER: THE GRAPH

10. HYPOTHESIS TESTING …
Fair dice Loaded dice
Accept H0 (do not accept H1) 35/36
Reject H0 (accept H1) 1/36
Therefore p = ?

11. P VALUE …
•p = 1/36 (0.02777…) which is smaller than
WHAT?
•Shall I choose 0.05?!?!
•Do you reject H0?

12. P VALUE …
•Here alpha is 0.025
•H0 dice are fair
•H1 dice are loaded so that 12 occurs more
often (one-sided test)!!

13. •FIRST KEY MESSAGE: p < 0.05 or 0.025 can
be observed by pure chance!!!

14. WHAT WOULD YOU DO NEXT TO INCREASE
YOUR CHANCES OF MAKING THE RIGHT
DECISION?

15. LET’S ASSUME THAT THE DICE ARE
LOADED …
• For this hypothesis, we will assume that IF the dice are loaded:
• The probability of obtaining 6 is 5/6
• Therefore, the probability of obtaining 1, 2, 3, 4 or 5 is 1/6 (1/30 each).
• Therefore the power of my test is 25/36 = 69%. Do I have enough power to accept the alternative
hypothesis i.e. reject the null hypothesis that the dice are not loaded?
Fair dice Loaded dice
Accept H0 (do not accept H1) 35/36 11/36 (= 1 – 25/36) i.e. all
combinations without 12 (2*6)
Reject H0 (accept H1) 1/36 25/36 (I can reject H0 only if
double 6 i.e. 12 is obtained)

16. WHAT SHALL I DO NEXT??

17. INCREASE THE SAMPLE SIZE …
PERFORM A SECOND THROW …
Fair dice Loaded dice
Accept H0 1 - (1/36)2 (11/36)2: I do not have any
evidence to reject H0 if 12
(double 6) is not obtained
I throw the dice twice …
Reject H0 (1/36)2: I can safely reject the
null hypothesis if double 6 was
obtained twice …
1 – (11/36)2: I can safely reject
H0 only if at least one double
i.e. 12 is obtained in 2 throws …
Therefore power = 1 – (11/36)2 = 90%

18. DO I REJECT THE NULL HYPOTHESIS?
OR
DO I ACCEPT THE ALTERNATIVE
HYPOTHESIS?

19. TO ANSWER THIS QUESTION, PLEASE
CONSIDER THE FOLLOWING …
AND SHE SMILES …

20. DO I REJECT THE NULL HYPOTHESIS?
OR
DO I ACCEPT THE ALTERNATIVE
HYPOTHESIS?

21. LET’S ASSUME THAT THE TRICK IS LESS
OBVIOUS, THE DICE ARE SLIGHTLY LESS
LOADED …• For this hypothesis, we will assume that the dice are loaded so that:
• The probability of obtaining 6 is 1/2 (instead of 5/6 in the previous example).
• Therefore, the probability of obtaining 1, 2, 3, 4 or 5 is 1/2 (1/10 each).
• Therefore the power of my test is 1/4 = 25%. Do I have enough power to accept the alternative
hypothesis i.e. reject the null hypothesis?
Fair dice Loaded dice
Accept H0 (do not accept H1) 35/36 3/4
Reject H0 (accept H1) 1/36 1/4 (I can reject H0 only if
double 6 i.e. 12 is obtained)

22. INCREASE THE SAMPLE SIZE …
PERFORM A SECOND THROW …
Fair dice Loaded dice
Accept H0 1 - (1/36)2 (3/4)2: I do not have any
evidence to reject H0 if 12
(double 6) is not obtained
2 throws …
Reject H0 (1/36)2: I can safely reject the
null hypothesis if double 6 was
obtained twice …
1 – (3/4)2: I can safely reject H0
only if at least one double 6
12 is obtained in 2 throws …
Therefore power = 1 – (3/4)2 = 44%

23. INCREASE THE SAMPLE SIZE …
PERFORM A THIRD THROW …
Fair dice Loaded dice
Accept H0 1 - (1/36)3 (3/4)3: I do not have any
evidence to reject H0 if 12
(double 6) is not obtained
3 throws …
Reject H0 (1/36)3: I can safely reject the
null hypothesis if double 6 was
obtained 3 times …
1 – (3/4)3: I can safely reject H0
only if at least 1 double 6 i.e. 12
is obtained in 3 throws …
Therefore power = 1 – (3/4)3 = 59%

24. POWER OF 80% ONLY ACHIEVED
AFTER 6 THROWS …
• The power of my test after 6 throws is: 1 – (3/4)6 = 82%

25. QUIZ TIME
I am the best. My study is powered at
90%.
□ Correct
□ Incorrect

26. QUIZ TIME
I am the best. My study is powered at
90%.
□ Correct
□ Incorrect √ (the power refers to a
statistical test – no to a study).

27. QUIZ TIME
If the power of your statistical test if too low,
you may …
□ Wrongly accept a product which is ineffective
□ Wrongly reject a product which is effective …

28. QUIZ TIME
If the power of your statistical test if too low,
you may …
□ Wrongly accept a product which is ineffective
□ Wrongly reject a product which is effective √
… (you do not have the power to reject H0 i.e.
accept H1).

29. QUIZ TIME
For my test and a given sample size, the
risks alpha and beta are unrelated (do not
influence) each other …
□ True
□ False

30. QUIZ TIME
For my test and a given sample size, the
risks alpha and beta are unrelated (do not
influence) each other …
□ True
□ False √ (see power graph)

31. QUIZ TIME
But I can choose the power of my test …
□ True
□ False

32. QUIZ TIME
But I can choose the power of my test …
□ True √
□ False

33. QUIZ TIME
In a trial when I perform multiple tests, I
inflate the risk …
□ Alpha
□ Beta

34. QUIZ TIME
In a trial when I perform multiple tests, I
inflate the risk …
□ Alpha √
□ Beta

35. QUIZ TIME
For trial JB007, we are planning to perform multiple testing
comparisons and some multiplicity adjustments all these
comparisons, could you advise me whether this multiplicity
adjustment might have an impact on the power of the test of
my primary hypothesis?
□ The multiplicity adjustment may have an impact on the power
□ Multiplicity adjustments never impact statistical power …

36. QUIZ TIME
For trial JB007, we are planning to perform multiple testing
comparisons and some multiplicity adjustments all these
comparisons, could you advise me whether this multiplicity
adjustment might have an impact on the power of the test of
my primary hypothesis?
□ The multiplicity adjustment may have an impact on the power
√ (since it could change the alpha threshold under which I
reject H0).
□ Multiplicity adjustments never impact statistical power …

37. QUIZ TIME
For trial JB007, we are planning to perform a total of 40 testing
procedures and perform a Bonferroni correction (i.e. use a
alpha value of 0.05/40) could that have an impact on the
statistical analysis?
□ You may inflate the risk of type 1 error (reject the null
hypothesis even if it is true)
□ Who needs effective medicines?…
□ Keep calm and carry on …

38. QUIZ TIME
For trial JB007, we are planning to perform a total of 40 testing
procedures and perform a Bonferroni correction (i.e. use a
alpha value of 0.05/40) could that have an impact on the
statistical analysis?
□ You may inflate the risk of type 1 error (reject the null
hypothesis even if it is true)
□ Who needs effective medicines?… √ (see power graph).
□ Keep calm and carry on …

39. QUIZ TIME
For trial JB007, based on our primary hypothesis the
estimated sample size is 100 patients, we are
planning to enrol 500 patients instead in our trial. Is
that a problem?
□ Yes
□ No!
□ Errr …

40. QUIZ TIME
For trial JB007, based on our primary hypothesis the
estimated sample size is 100 patients, we are planning to
enrol 500 patients instead in our trial. Is that a problem?
□ Yes √ (You could achieve significance for an effect size
which is smaller than the effect of your primary
hypothesis).
□ No!
□ Errr …

41. QUIZ TIME
What are the factors which can influence the power of a test?
□ Sample size
□ Multiplicity adjustments
□ Randomisation scheme (e.g. 1:1 vs 1:2)
□ Interim analyses
□ Missing data or drop-out rate (or lost to follow-up)
□ The sponsor

42. QUIZ TIME
What are the factors which can influence the power of a test?
□ Sample size
□ Multiplicity adjustments
□ Randomisation scheme (e.g. 1:1 vs 1:2)
□ Interim analyses
□ Missing data or drop-out rate (or lost to follow-up)
□ The sponsor
ANSWER: ALL OF THE ABOVE

43. CONCLUSIONS
• Do not forget that p < 0.05 or 0.025 can be observed by pure chance …
• A test is based on type I but also on type II error (power)
• The performance of the test (hence power) depends on your hypotheses
• The size and power of your experiment will depend on the magnitude and
variability of the difference that you try to observe (the higher the difference,
the easier it will be to establish a difference) (NB: and your risk alpha which is
usually set up at 0.05).
• The sample size will increase the power of your test and will allow you to
achieve the statistical power that you want to achieve.
• Many other factors can influence the power of a test, this includes: multiplicity
adjustments, interim analyses, etc.
• A smile does not play any role in hypothesis testing.