The document discusses effective strategies for A/B testing, emphasizing the importance of setting clear goals, achieving statistical significance, and understanding the full marketing funnel rather than just click conversions. It highlights Kissmetrics' approach to A/B testing, which involves substantial sample sizes and a careful analysis of data over time. The webinar features insights from marketing experts Tomasz Borys and Thue Madsen, focusing on making data-driven decisions in marketing operations.

1. A/B Testing
You Might Be Driving in the Wrong Direction
TOMASZ BORYS

2. @NEILPATEL
@Kissmetrics
#KissWebinar
@thuelmadsen

3. Thue is the Kissmetrics Webinar Wizard and
Marketing Ops Manager. Before joining forces with
Kissmetrics, he was a Lyft driver in SF, which is also
how he ended up as a KISSmetrics marketer.
Whenever Thue is not trying to automate everything
around him, you can find him hiking in the Sierras.
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
Tomasz loves dipping his feet in the river while
fishing, injuring his thumb while gaming, hacking
away at a golf club, and driving demand at
Kissmetrics.
He’s also the biggest fan of gummy bears.
TOMASZ BORYS
Director of Marketing, Kissmetrics
@tbcali

4. @NEILPATEL
@Kissmetrics
#KissWebinar
@tbcali

5. 1 The optimum strategies for A/B testing
2 How we A/B Test at KISSmetrics
3
A/B Testing beyond click conversions - having a pulse on
the entire funnel and why it’s important
TABLE OF CONTENTS
4 The influence social traffic has in the funnel

6. WATCH WEBINAR RECORDING NOW

7. • Set a Goal
• Baby Steps
• Aim for Statistical Significance
• Never Lose Sight
THE OPTIMUM STRATEGIES FOR A/B TESTING

8. Set a Goal

9. Baby Steps
People want to move the
needle and see results fast, by
implementing multiple changes
at once.
But how can you be sure what
element had an impact?

10. Aim for Statistical Significance
SET A GOAL
Aimfor Statistical
Significance
getdatadriven.com

11. Never Lose
Sight
We can caught up on
what we think is best
to optimize
conversions or
traffic…but data
doesn’t lie

12. • 4,000 and above sample size
• 99% Statistical Confidence
• Blind Eye for 1 Week
HOW WE A/B TEST AT KISSMETRICS

13. 4,000 and Above Sample Size
Significant or decent amount of data is needed
when you’re testing beyond click conversions

14. 4,000 and Above Sample Size
significant or decent amount of data is needed when
you’re testing beyond click conversions
99% Statistical
Confidence
Police

15. Blind Eye
for 1 Week
Data can be very erratic the
first several days, so it’s
easy to hit the panic button

16. A/B Testing Beyond Click
Conversions

17. ORIGINAL

18. VARIANT

19. SIGNUPS

20. SIGNUPS

21. SIGNUPS

22. Signup Install JS
Custom
Data
Opps

24. Signups
Installed JS

25. Signups
Install JS

26. Signups
Install JS

27. Custom Data

28. Custom Data

29. Custom Data

30. Custom Data
Opportunities Created

31. Custom Data
Opportunities Created

32. Custom Data
Opportunities Created

33. SOCIAL AD FUNNEL

34. LOG IN WITH GOOGLE
Start Your Free KISSmetrics Trial

35. TOMASZ BORYS
Director of Marketing, Kissmetrics
@tbcali
tborys@kissmetrics.com
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
tmadsen@kissmetrics.com
Questions?