How to calculate the statistical significance of your A/B testing in Excel.

1. QUANTITATIVE & QUALITATIVE ANALYSIS
HOW TO
KNOW YOUR
A/B TESTING
STATISTICAL SIGNIFICANCE
https://www.linkedin.com/in/lilimarialau
@LiliMariaLau

2. • Extreme Sports runner
• Painter: Sketch, Watercolor, Oil, Pastels, Charcoal, Origami
• Qi-Gong practitioner
Lili Maria Lau
MSF, MIB
linkedin.com/in/lilimarialau
@LiliMariaLau
HELLO!

3. KNOW YOUR
A/B TESTING
STATISTICAL SIGNIFICANCE
1 Methodology
2 Analysis
3 Conclusion & Recommendation

4. OBJECTIVE:
• Determine which call to action button:
is more effective at converting potential
customers?
DATA:
• 600 visits
• 300 shown button
• 300 shown button
• 0 = didn't click it
• 1 = clicked it
METHODOLOGY
1 1 1
2 0 1
3 1 1
. . .
. . .
. . .
298 0 0
299 1 0
300 1 1
Total 142 156

5. METHODOLOGY: Chi Square 2X2 contingency table
𝑋"
=
(𝑎𝑑 − 𝑏𝑐)"
∗ (𝑎 + 𝑏 + 𝑐 + 𝑑)
(𝑎 + 𝑏)(𝑐 + 𝑑)(𝑏 + 𝑑)(𝑎 + 𝑐)
𝑋" =
[ 142 ∗ 144) − (158 ∗ 156 ]"∗ (142 + 158 + 156 + 144)
(142 + 158)(156 + 144)(158 + 144)(142 + 156)
𝑋" =
(20,448 − 24,648)"∗ (600)
(300)(300)(302)(298)
𝑋" = 1.307
Formula to calculate Chi Square:
𝑯 𝟎: 𝑇ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑙𝑖𝑐𝑘𝑠 𝑖𝑠 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡 𝑓𝑟𝑜𝑚 𝑡ℎ𝑒 𝑐𝑜𝑙𝑜𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑏𝑢𝑡𝑡𝑜𝑛
𝑯 𝟏: 𝑇ℎ𝑒 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑙𝑖𝑐𝑘𝑠 𝑖𝑠 𝑎𝑠𝑠𝑜𝑐𝑖𝑎𝑡𝑒𝑑 𝑤𝑖𝑡ℎ 𝑡ℎ𝑒 𝑐𝑜𝑙𝑜𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑏𝑢𝑡𝑡𝑜𝑛
Downloaded
Didn't
download Total
142 158 300
156 144 300
Total 298 302 600
Downloaded
Didn't
download Total
a b a + b
c d c + d
Total a + c b + d a + b + c + d = N

6. Degrees of freedom = (# of columns - 1) x (# of rows - 1)
Therefore, df = (2 - 1) x (2 - 1) = 1
𝑋"
= 1.307
𝛼 = 0.05
df = 1
In Excel we calculate the Chi-Square formula with
X = 1.307, and deg_freedom = 1, to find the p-value:
P value = CHISQ.DIST.RT(1.307,1)
P value = 0.253
METHODOLOGY: Chi-Square 2x2 contingency table

7. ANALYSIS
Since a p-value of 0.253 is greater than the conventionally accepted
significance level of 0.05 (i.e. p > 0.05), we fail to reject the null
hypothesis.
In other words, there is no statistically significant difference or
relationship between the number of downloads, and whether the button is
or .

8. There is no relationship between
the number of clicks, and the
button design ( or )
with statistical significance.
“
CONCLUSION
“

9. SUGGESTED NEXT STEPS
Try other tools that would have more impact in the number of downloads taking
into account the following:
• Determine who is interesting in buying in your website and their specific
characteristics
• Define your site content and keywords to ranking high in the potential
searches that those users might make
• Identify themes in customers that can be applied to those keywords
• Group your customers into buyer "personas"
• Target those personas that are most likely to make a purchase in your
website not just visit!

10. THANKS FOR READING!