A/B Testing 101
with JR Hopwood

A

B
Overview
Introduction

Need convincing?
Deciding on a test
Evaluate a test
Wrap up

A B
Vs.
Introduction
About the presenter

JR Hopwood
 With over a decade of web analytics and website testing experience, JR works with iAPPS
...
Bridgeline Digital iAPPS Success Group
Need Convincing?
Need Convincing?

Do you like peanut butter?

Do you like jelly?

Do you like PB&J?
Need Convincing?
You just tested!




Peanut Butter has always worked for you…
But you wondered if Jelly could add some...
Need Convincing?

AB Testing allows you to keep what is working
today and just test a subset of traffic
All Visitors

75%
...
Need Convincing?

Now what if that translated to more
conversions?
All Visitors
If each conversion
was $1 and you
had 1,00...
Need Convincing?

Conversely, what if that translated to less
conversions?
All Visitors

75%

25%

5% conversion
Current P...
Need Convincing?
Stop the bleeding!
 Luckily with AB Testing you can turn it off…
 … just try and wait until there is so...
Need Convincing?
Improve ROI or minimize the losses


Testing can only help conversion in the long run
 Even a failed te...
Need Convincing?
Here‟s an example scenario I see all the time…




Page A converts at 25% and has for several months…
...
Decide what to test
Decide what to test
Define a Goal
The How‟s





Gut / Common Sense
Web Analytics
Settle an argument…
I‟m not kidding
...
Decide what to test
Don‟t just test to test…



Make sure you are staying true to your KPIs
Whether you sell widgets or ...
Decide What to Test | Gut / Common Sense
Why test a page you just feel doesn‟t have the best usability?
 GUT…

Why NOT te...
Decide what to test | Gut / Common Sense
Some questions to consider:
 Where is the largest fallout / drop off / abandonme...
Decide What to Test | Web Analytics
Numbers don‟t lie
 … often
 Before making decisions based on your analytics please b...
Decide What to Test | Web Analytics
Some questions to consider:
 Are there pages that have a higher bounce rate than most...
Decide What to Test | Settle an argument
Marketing person A says one thing and Marketing person B says another
 Test it a...
Decide What to Test | Design
Obviously design changes are the low hanging fruit of website testing
•

Huge gains have been...
Decide what to test | Functionality
There are so many ways to code a widget or functionality on a website
•
•

Think about...
Decide What to Test | Content
It is amazing how the smallest content changes can make a difference
•

•

What would you cl...
Evaluate a Test

Evaluate a test
Evaluate a Test
Testing platforms come with evaluation tools out of the box, use them!
•
•

The metric you‟re looking for ...
Evaluate a Test
The higher the Significance the more confident you can be to move forward with that
variation
•
•

Obvious...
Things to Consider for your First Test
Don‟t go overboard, but be bold
•
•
•

Start with elements that are easy to spot
Do...
First Test Checklist
Have I defined my website goals?
Have I evaluated where a problem may exist?
Have I come up with a so...
Now it‟s your turn to become
the mad scientist!

Q&A
Thank You!

this webinar prepared by
the iSG Digital Services Team
at Bridgeline Digital
For more information contact iSG ...
Upcoming SlideShare
Loading in...5
×

iSG Webinar – AB Testing: The most important thing you’re NOT doing

546

Published on

AB Testing requires a considerable amount of time, work, and planning. But as with most ventures that involve those components, the valuable results and insight gained from AB Testing more than payoff the initial investment.

Watch a recording of a webinar with JR Hopwood from our iAPPS Success Group (iSG) to learn about the process of AB Testing game and how to convince others in your organization to AB Test as well.

Published in: Marketing
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
546
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

iSG Webinar – AB Testing: The most important thing you’re NOT doing

  1. 1. A/B Testing 101 with JR Hopwood A B
  2. 2. Overview Introduction Need convincing? Deciding on a test Evaluate a test Wrap up A B Vs.
  3. 3. Introduction
  4. 4. About the presenter JR Hopwood  With over a decade of web analytics and website testing experience, JR works with iAPPS website partners to assist them in getting the most out of their web applications. The iAPPS Success Group (iSG) is a team of support and digital services specialists who help Bridgeline Digital customers succeed online.
  5. 5. Bridgeline Digital iAPPS Success Group
  6. 6. Need Convincing?
  7. 7. Need Convincing? Do you like peanut butter? Do you like jelly? Do you like PB&J?
  8. 8. Need Convincing? You just tested!    Peanut Butter has always worked for you… But you wondered if Jelly could add some spice… And then, WHOA, what about adding them together??? A/B Testing works the same way   We take something that may be „working‟ today and see if we can improve on it Sometimes we‟re right, sometimes we‟re wrong A/B Testing provides the numbers   You no longer have to rely solely on opinions Testing analytics will show you precisely what works and what doesn‟t
  9. 9. Need Convincing? AB Testing allows you to keep what is working today and just test a subset of traffic All Visitors 75% 25% Current Page Test Page Ways to create subsets: Percentage Traffic Source Site Action Etc…
  10. 10. Need Convincing? Now what if that translated to more conversions? All Visitors If each conversion was $1 and you had 1,000 visits you made $12.50 more than you would have! 75% 25% 5% conversion Current Page 10% conversion (+100%) Test Page
  11. 11. Need Convincing? Conversely, what if that translated to less conversions? All Visitors 75% 25% 5% conversion Current Page If each conversion was $1 and you had 1,000 visits you only lost $6.25 over what you would have made… you tried. 2.5% conversion (-50%) Test Page
  12. 12. Need Convincing? Stop the bleeding!  Luckily with AB Testing you can turn it off…  … just try and wait until there is some statistical significance  Sometimes conversion numbers are skewed due to visitor behavior changes, so be patient  Example: No one buys on the weekends Statistical Significance = Confidence
  13. 13. Need Convincing? Improve ROI or minimize the losses  Testing can only help conversion in the long run  Even a failed test can prove to you that you SHOULD NEVER do that variation again  Since you aren‟t sending all of your traffic to the new variation the gains and losses have a lessor effect on your current bottom line  …but could provide a substantial change to your future one!  This gives you the opportunity to try new things without going ALL IN  Just remember, you need enough traffic to gain Statistical Significance
  14. 14. Need Convincing? Here‟s an example scenario I see all the time…    Page A converts at 25% and has for several months… Marketing team doesn‟t want to touch it, because… CEO doesn‟t want to lose revenue… What is a marketer to do?  Remind CEO that we aren‟t changing the page for everyone  Do the analysis… show that even a 3% bump in conversion on that page translates into $n increase in revenue  Calculate the resource time of those involved (including agencies)  Compare this against the potential gains… and you‟ll get the green light  Even if your site is non-revenue driving still show conversion increases  Show how these translate into real people  If your site just pushes promotional brochure-ware, but for each brochure downloaded your company makes $n in return… get the picture?  CEO‟s like the bottom line  Once you prove that testing can have an immediate impact on bottom line the tide will turn and your CEO will start asking YOU when the next test is
  15. 15. Decide what to test
  16. 16. Decide what to test Define a Goal The How‟s     Gut / Common Sense Web Analytics Settle an argument… I‟m not kidding The What‟s  Design  Functionality  Content
  17. 17. Decide what to test Don‟t just test to test…   Make sure you are staying true to your KPIs Whether you sell widgets or provide information you have something to test We want to test because we want to…     Increase revenue Increase conversion Decrease Bounce Rate Decrease Exits Once your goal is defined the fun begins…
  18. 18. Decide What to Test | Gut / Common Sense Why test a page you just feel doesn‟t have the best usability?  GUT… Why NOT test a page that isn‟t included in your funnel and that no one?  COMMON SENSE… It‟s OK to use these as a basis for a test
  19. 19. Decide what to test | Gut / Common Sense Some questions to consider:  Where is the largest fallout / drop off / abandonment?  What page presents the largest impact to conversion?  Does a page not match your brand?  Does a landing page not match the offer?
  20. 20. Decide What to Test | Web Analytics Numbers don‟t lie  … often  Before making decisions based on your analytics please be sure it is implemented properly! Metrics to consider  Bounce Rates  Exit Rates  Funnel Abandonment
  21. 21. Decide What to Test | Web Analytics Some questions to consider:  Are there pages that have a higher bounce rate than most?  Are there pages that have a higher exit rate than most?  Does a page in a funnel have an unusually high abandonment? A little gut…  These are great questions, but sometimes there are explanations…  Does an exit link to off site?  Do you push visitors to call or email?  Does the page have an iFrame, Flash or HTML 5 element?
  22. 22. Decide What to Test | Settle an argument Marketing person A says one thing and Marketing person B says another  Test it and prove which one does better!
  23. 23. Decide What to Test | Design Obviously design changes are the low hanging fruit of website testing • Huge gains have been achieved by testing: • Buttons • Imagery • Layout Don‟t go overboard… • The more changes you make to a page the less specific the test can be • Example: If you wanted to test a form location on a landing page to see if the left or right side converted better (could be a great test) • However, if at the same time you change the button color or copy… • What actually tested better? The layout change or the other elements?? Multivariate also has a time and a place • • Multivariate testing allows for multiple changes to layout and sections on a web page at one time • This can get tricky if you don‟t have enough traffic OR if your tests are WILDLY different My recommendation would be to only move to multivariate after a few AB Tests so you get the hang of it and what to look for in the evaluation
  24. 24. Decide what to test | Functionality There are so many ways to code a widget or functionality on a website • • Think about how many Date Pickers you‟ve seen out there… Sometimes though there is a need to test these • In our Date Picker example, is showing 2-months better than just showing 1? • What about a login, should you ask for user name and password at once or just user name, then password on the next page? Get layout right first… • Before you turn to functionality as the problem are you doing a good enough job explaining or calling out on the page?
  25. 25. Decide What to Test | Content It is amazing how the smallest content changes can make a difference • • What would you click on more? • “Begin Reservation” • “Continue” What content would you engage with more? • “You really need this product” • “We really think you should have this product” Don‟t couple with design… • • As mentioned previously, if you change the layout or button colors you won‟t be able to tell for sure whether it was the copy or the other changes! If you must do this at the same time, do a multivariate test so you can see how the variations perform independently AND together
  26. 26. Evaluate a Test Evaluate a test
  27. 27. Evaluate a Test Testing platforms come with evaluation tools out of the box, use them! • • The metric you‟re looking for is Statistical Significance, Confidence or Probability to Beat Original What this shows is if the variation were to run 24/7/365 what is the chance it will outperform the control, or original • Why this is necessary is that in the test the page actually ISN‟T being shown to every visitor • Statistical analysis crunches the numbers and determines the probability based on various factors
  28. 28. Evaluate a Test The higher the Significance the more confident you can be to move forward with that variation • • Obviously the more visitors that conduct the test and perform better on the competing variation the higher that significance will be Example: If you throw me 10 pitches and I hit 7, you can be somewhat sure I have about a 70% chance of hitting the next pitch. But if you throw me 100 pitches and I hit 70 you can be more confident in that 70%, since it has been confirmed over a longer period and by several pitches DO NOT stop a test if Statistical Significance has not been reached • This only skews the data set and may not play out once you go to 100%
  29. 29. Things to Consider for your First Test Don‟t go overboard, but be bold • • • Start with elements that are easy to spot Don‟t let your first test simply be copy changes The reason for this is so in your first test you‟ll be able to see bigger variations in conversion data and thus be able to learn about test confidence Sometimes the end is better… • • • If you lose 10% of your visitors on the final step of your booking flow just think how quickly your ROI could change if you lost only 5%... Or less Sometimes small changes on those final pages are the difference between clicking “BILL ME” and exiting These visitors made it all the way to the final step… they didn‟t leave because they are no longer interested! Have FUN! • • • • Get other departments involved to secure buy in “Hey design team, if you could redesign this page what would it look like?” “Hey sales team, what would you change to make the site better?” • Warning, this team is competitive!!!! With A/B Testing the numbers don‟t lie… a variation wins and variations lose
  30. 30. First Test Checklist Have I defined my website goals? Have I evaluated where a problem may exist? Have I come up with a solid competing variation(s)? Have I come up with a hypothesis on which variation will win? Have I devised a segment strategy for who sees the test variations? Is one variation outperforming another? Do we have statistical significance? Go with that variation Repeat
  31. 31. Now it‟s your turn to become the mad scientist! Q&A
  32. 32. Thank You! this webinar prepared by the iSG Digital Services Team at Bridgeline Digital For more information contact iSG at: strategysupport@blinedigital.com Or call Seth Rich : (781) 404-5423
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×