As marketers and businesses in general, can any of us truly say that we are using the absolute best version of every marketing tactic in our arsenal? I think not – and those that can are few and far between. For the majority of marketing’s lifetime as a professional discipline, decisions, content direction, design, and strategy have largely been influenced by opinion and gut feel. While going with our gut can create a good jumping off point, it is not the key ingredient for a solid, ongoing marketing strategy. Do any of the quotes below sound familiar? “This email subject line should work best.” “I think it should be red instead of blue.” “How many form fields should we use?” “This content might resonate with our audience better than that.” “Are you sure we should say Free Demo instead of Complimentary Consultation?” “What should we try to improve conversion rates? These are just a few examples among thousands that we experience everyday, and that is exactly why we need to utilize a scientific approach to take the guesswork out of marketing. Luckily there is a simple method at our disposal that can provide clear direction, continuous improvement, and a real positive impact on the bottom line of your business – A/B split testing. DOYO Live proudly announces that we’ve added Alex Foley to our speaker lineup. In this engaging presentation, Alex will walk you through the theory and history behind A/B split testing, where to use it in your business, best practices, and some tools available to help kick-start your new marketing best friend. As a Consultant at NR Media Group, Alex manages multiple accounts and helps drive inbound marketing and HubSpot strategy/execution for his clients. As an inbound and marketing automation specialist, Alex understands the strategies necessary to succeed in today’s evolving business and consumer landscape. In addition to his primary client responsibilities, he also assists with NR Media Group’s own marketing efforts and regularly contributes to the blog. Alex graduated from Ohio State’s Fisher College of Business with focuses in Finance and Professional Writing and has over 6 years of digital marketing experience. In the past, Alex has helped to create strategic content and product-launch strategies for numerous Fortune 100 brands including Sprint and Humana. In his free time, Alex loves to play sand volleyball, embarks on scuba diving adventures and frets over the fate of all the Game of Thrones characters that he connects with. Connect with Alex on LinkedIn and Twitter. DOYO Live - A digital marketing and interactive design conference will take place on August 3, 2017 in Youngstown, Ohio!

1. The Power of A/B Split Testing
Presentation By: Alex Foley
August 4, 2016

2. My pain point:
“I’m not sure how to best open up
my presentation at #DOYOLive.”

3. HYPOTHESIS 1: ENTHUSIASTIC, CONFIDENT; WWE TYPE
My hypothesis:
The crowd may need to be energized!
Pro:
Crowd attention and energy
Con:
Potentially make a fool of myself

4. FUN FACT…

5. HYPOTHESIS 2: RIDE THE POP CULTURE WAVE
My hypothesis:
A Pokemon Go reference can’t
fail, right?
Pro:
Widely known and top of mind
Con:
Some folks may already be sick
of the Poke-mania

6. The Results?

7. VOTE ON MY NEXT TEST
• Justin Bieber
• Drill Instructor
• Trump

9. WHAT WE WILL LEARN
What is it?
Why it’s important?
Best Practices
The New Wave of Website Development
Resources

10. How Marketing “Used
to” Be Done

11. HOW MARKETING “USED TO” BE DONE
Source: isha.sadguru.org

12. 87% of Marketers Consider Data
Their Most Underutilized Asset.
-Teradata 2015 Global Data-Driven Marketing Survey

13. What is A/B Split Testing?

14. A/B Split Testing: testing a hypothesis by measuring the
performance of one version (A) against another (B) in an equal
sample size.

15. In reality – it is the scientific method

16. Why is this important
for us as marketers?

17. It keeps you thinking, it keeps you sharp
It is the formula for continuous improvement
You don’t have to be right the first time
It can give you an edge on the competition

18. It directly impacts the bottom line
74% of companies who take a structured approach to
conversion (including A/B testing) have improved sales
Source: Econsultancy and
RedEye 2014

19. Modern Origins

21. TAGUCHI’S LOSS FUNCTION—A CAR PART EXAMPLE
Bad Car Part
Take Car to
Dealer
Dealer
Staffs
Accordingly
Time
Wasted on
Preventable
Problem
Missing
Work to
Deal with
Repairs

22. YOU MIGHT SAY THIS IS… THE BUTTERFREE EFFECT!

23. THE FOUNDATION OF TODAY’S LEADING COMPANIES

24. THE BASIC A/B TESTING METHODOLOGY
A/B
Learnings
Define
Goals and
Hypothesis
Test a
variable
Analyze
and
Conclude
Record
Findings

25. The Golden Rule of A/B Testing:
Only test 1 variable at a time!

26. WHAT SHOULD I BE TESTING?
• Email subject lines
• Email preview text
• CTAs
• Colors
• Size
• Copy
• Lengths of copy
• Types
• Sales copy
• Buy now vs. learn more vs. x
• Headlines
• Font
• Font color
• Bullets vs. no bullets
• Amount of content
• Amount of choices or CTAs
• Page templates
• Layouts
• Heavy images or video vs. text
• Email personalization tokens
• Pricing
• Pricing copy
• Form aesthetics
• Images – size, positioning, type, #
• Email sending times
• Email senders
• Persona-based content
• Number of form fields
• Form field wording

27. So…everything?

28. THE CONUNDRUM – WHERE TO START!?
• Start with your goals and identify primary
influencers
• Start with your leading hypotheses
• Start with key assets that underperform
• Start big and go narrow

29. A/B Testing —the Basis
for Growth Driven Design

30. TRADITIONAL WEB DESIGN
1.5 - 2 Years
1.5 - 2 Years
3 Month
Redesign
Legacy
Website
Project Budget
$25,000 - $250,000
3 Month
Redesign
Website
V01
Website
V02
Project Budget
$25,000 - $250,000
Impact
Time

31. THE GDD DIFFERENCE
Impact
1.5 - 2 Years
1.5 - 2 Years
3 Month
Redesign
Launch
Pad
Website
Project Budget
$25,000 - $250,000
3 Month
Redesign
Time
Website
V01
Website
V02
Project Budget
$25,000 - $250,000
Gained Opportunity
Gained Opportunity
11/29/201
6
31

32. LOOK AT YOUR TOP-VIEWED PAGES

33. BRAINSTORM AND HYPOTHESIZE

34. UTILIZE HEATMAPS AND USER SESSION DATA

35. UTILIZE HEATMAPS AND USER SESSION DATA

36. A/B Testing Software

37. A/B TESTING TOOLS

38. A/B Testing Without
Tools

39. It is an imperfect science, but it is quite doable!

40. MANUAL TESTING

41. CREATE A TRACKING SPREADSHEET

42. HOW LONG IS LONG ENOUGH?
It depends on:
• Overall traffic
• Traffic to that asset
• The scale of data for your business
Pro Tip
Test it!
Find your benchmarks.

43. USE A CONFIDENCE CALCULATOR

44. A Client Case Study

45. Our Hypothesis: 3D book covers will drive more clicks
than flat book covers

46. Flat Image 3D Image
Success Metric: % of views to clicks

47. THE RESULTS MAY SURPRISE YOU
Slight difference in views to clicks
Bigger difference in clicks to submissions

48. An A/B Testing Case Study –
Sim City

49. AN A/B TESTING CASE STUDY – SIM CITY

50. MULTIVARIATE TESTING VS. A/B TESTING
A/B = testing 1 variable
Multivariate testing = testing many
variables and combinations of those
Ideal for high traffic websites
Source: optimizely.com

51. A MULTIVARIATE CASE STUDY – 2008 PRESIDENTIAL CAMPAIGN
Success metric: sign-up rates
24 total combinations: 4 button – 6 image
multivariate test
Hypothesis: video media elements will
perform better
Source: blog.optimizely.com

52. A MULTIVARIATE CASE STUDY – 2008 PRESIDENTIAL CAMPAIGN
The Results
• Hypothesis disproved
• 40.6% improvement rate
• 2,880,000 additional emails
• $60 million in donations
Source: blog.optimizely.com

53. A REVIEW OF KEY CONCEPTS
• A/B testing is not an option, it is a must to stay competitive
• Only test one factor at a time per hypothesis
• You must be patient and wait for a significant sample size before drawing
conclusions
• You are never done with A/B testing – don’t settle
• Track the results! Set regular times to update and revisit your findings
• Make it part of your culture and daily thought process

54. RESOURCES
• How to Test Your Landing Pages
infographic
• The A/B Test Calculator and A/B Testing
Guide from HubSpot
• The A/B Test Flow Chart – print this out and
hang it up!
• Hotjar – a free heat mapping tool
• 71 A/B Testing Ideas
• 10 A/B Testing Tools
• Growthdrivendesign.com
• http://hubs.ly/H03W7hV0

55. Thank You!