Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

iSG Webinar – AB Testing: The most important thing you’re NOT doing


Published on

AB Testing requires a considerable amount of time, work, and planning. But as with most ventures that involve those components, the valuable results and insight gained from AB Testing more than payoff …

AB Testing requires a considerable amount of time, work, and planning. But as with most ventures that involve those components, the valuable results and insight gained from AB Testing more than payoff the initial investment.

Watch a recording of a webinar with JR Hopwood from our iAPPS Success Group (iSG) to learn about the process of AB Testing game and how to convince others in your organization to AB Test as well.

Published in: Marketing
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. A/B Testing 101 with JR Hopwood A B
  • 2. Overview Introduction Need convincing? Deciding on a test Evaluate a test Wrap up A B Vs.
  • 3. Introduction
  • 4. About the presenter JR Hopwood  With over a decade of web analytics and website testing experience, JR works with iAPPS website partners to assist them in getting the most out of their web applications. The iAPPS Success Group (iSG) is a team of support and digital services specialists who help Bridgeline Digital customers succeed online.
  • 5. Bridgeline Digital iAPPS Success Group
  • 6. Need Convincing?
  • 7. Need Convincing? Do you like peanut butter? Do you like jelly? Do you like PB&J?
  • 8. Need Convincing? You just tested!    Peanut Butter has always worked for you… But you wondered if Jelly could add some spice… And then, WHOA, what about adding them together??? A/B Testing works the same way   We take something that may be „working‟ today and see if we can improve on it Sometimes we‟re right, sometimes we‟re wrong A/B Testing provides the numbers   You no longer have to rely solely on opinions Testing analytics will show you precisely what works and what doesn‟t
  • 9. Need Convincing? AB Testing allows you to keep what is working today and just test a subset of traffic All Visitors 75% 25% Current Page Test Page Ways to create subsets: Percentage Traffic Source Site Action Etc…
  • 10. Need Convincing? Now what if that translated to more conversions? All Visitors If each conversion was $1 and you had 1,000 visits you made $12.50 more than you would have! 75% 25% 5% conversion Current Page 10% conversion (+100%) Test Page
  • 11. Need Convincing? Conversely, what if that translated to less conversions? All Visitors 75% 25% 5% conversion Current Page If each conversion was $1 and you had 1,000 visits you only lost $6.25 over what you would have made… you tried. 2.5% conversion (-50%) Test Page
  • 12. Need Convincing? Stop the bleeding!  Luckily with AB Testing you can turn it off…  … just try and wait until there is some statistical significance  Sometimes conversion numbers are skewed due to visitor behavior changes, so be patient  Example: No one buys on the weekends Statistical Significance = Confidence
  • 13. Need Convincing? Improve ROI or minimize the losses  Testing can only help conversion in the long run  Even a failed test can prove to you that you SHOULD NEVER do that variation again  Since you aren‟t sending all of your traffic to the new variation the gains and losses have a lessor effect on your current bottom line  …but could provide a substantial change to your future one!  This gives you the opportunity to try new things without going ALL IN  Just remember, you need enough traffic to gain Statistical Significance
  • 14. Need Convincing? Here‟s an example scenario I see all the time…    Page A converts at 25% and has for several months… Marketing team doesn‟t want to touch it, because… CEO doesn‟t want to lose revenue… What is a marketer to do?  Remind CEO that we aren‟t changing the page for everyone  Do the analysis… show that even a 3% bump in conversion on that page translates into $n increase in revenue  Calculate the resource time of those involved (including agencies)  Compare this against the potential gains… and you‟ll get the green light  Even if your site is non-revenue driving still show conversion increases  Show how these translate into real people  If your site just pushes promotional brochure-ware, but for each brochure downloaded your company makes $n in return… get the picture?  CEO‟s like the bottom line  Once you prove that testing can have an immediate impact on bottom line the tide will turn and your CEO will start asking YOU when the next test is
  • 15. Decide what to test
  • 16. Decide what to test Define a Goal The How‟s     Gut / Common Sense Web Analytics Settle an argument… I‟m not kidding The What‟s  Design  Functionality  Content
  • 17. Decide what to test Don‟t just test to test…   Make sure you are staying true to your KPIs Whether you sell widgets or provide information you have something to test We want to test because we want to…     Increase revenue Increase conversion Decrease Bounce Rate Decrease Exits Once your goal is defined the fun begins…
  • 18. Decide What to Test | Gut / Common Sense Why test a page you just feel doesn‟t have the best usability?  GUT… Why NOT test a page that isn‟t included in your funnel and that no one?  COMMON SENSE… It‟s OK to use these as a basis for a test
  • 19. Decide what to test | Gut / Common Sense Some questions to consider:  Where is the largest fallout / drop off / abandonment?  What page presents the largest impact to conversion?  Does a page not match your brand?  Does a landing page not match the offer?
  • 20. Decide What to Test | Web Analytics Numbers don‟t lie  … often  Before making decisions based on your analytics please be sure it is implemented properly! Metrics to consider  Bounce Rates  Exit Rates  Funnel Abandonment
  • 21. Decide What to Test | Web Analytics Some questions to consider:  Are there pages that have a higher bounce rate than most?  Are there pages that have a higher exit rate than most?  Does a page in a funnel have an unusually high abandonment? A little gut…  These are great questions, but sometimes there are explanations…  Does an exit link to off site?  Do you push visitors to call or email?  Does the page have an iFrame, Flash or HTML 5 element?
  • 22. Decide What to Test | Settle an argument Marketing person A says one thing and Marketing person B says another  Test it and prove which one does better!
  • 23. Decide What to Test | Design Obviously design changes are the low hanging fruit of website testing • Huge gains have been achieved by testing: • Buttons • Imagery • Layout Don‟t go overboard… • The more changes you make to a page the less specific the test can be • Example: If you wanted to test a form location on a landing page to see if the left or right side converted better (could be a great test) • However, if at the same time you change the button color or copy… • What actually tested better? The layout change or the other elements?? Multivariate also has a time and a place • • Multivariate testing allows for multiple changes to layout and sections on a web page at one time • This can get tricky if you don‟t have enough traffic OR if your tests are WILDLY different My recommendation would be to only move to multivariate after a few AB Tests so you get the hang of it and what to look for in the evaluation
  • 24. Decide what to test | Functionality There are so many ways to code a widget or functionality on a website • • Think about how many Date Pickers you‟ve seen out there… Sometimes though there is a need to test these • In our Date Picker example, is showing 2-months better than just showing 1? • What about a login, should you ask for user name and password at once or just user name, then password on the next page? Get layout right first… • Before you turn to functionality as the problem are you doing a good enough job explaining or calling out on the page?
  • 25. Decide What to Test | Content It is amazing how the smallest content changes can make a difference • • What would you click on more? • “Begin Reservation” • “Continue” What content would you engage with more? • “You really need this product” • “We really think you should have this product” Don‟t couple with design… • • As mentioned previously, if you change the layout or button colors you won‟t be able to tell for sure whether it was the copy or the other changes! If you must do this at the same time, do a multivariate test so you can see how the variations perform independently AND together
  • 26. Evaluate a Test Evaluate a test
  • 27. Evaluate a Test Testing platforms come with evaluation tools out of the box, use them! • • The metric you‟re looking for is Statistical Significance, Confidence or Probability to Beat Original What this shows is if the variation were to run 24/7/365 what is the chance it will outperform the control, or original • Why this is necessary is that in the test the page actually ISN‟T being shown to every visitor • Statistical analysis crunches the numbers and determines the probability based on various factors
  • 28. Evaluate a Test The higher the Significance the more confident you can be to move forward with that variation • • Obviously the more visitors that conduct the test and perform better on the competing variation the higher that significance will be Example: If you throw me 10 pitches and I hit 7, you can be somewhat sure I have about a 70% chance of hitting the next pitch. But if you throw me 100 pitches and I hit 70 you can be more confident in that 70%, since it has been confirmed over a longer period and by several pitches DO NOT stop a test if Statistical Significance has not been reached • This only skews the data set and may not play out once you go to 100%
  • 29. Things to Consider for your First Test Don‟t go overboard, but be bold • • • Start with elements that are easy to spot Don‟t let your first test simply be copy changes The reason for this is so in your first test you‟ll be able to see bigger variations in conversion data and thus be able to learn about test confidence Sometimes the end is better… • • • If you lose 10% of your visitors on the final step of your booking flow just think how quickly your ROI could change if you lost only 5%... Or less Sometimes small changes on those final pages are the difference between clicking “BILL ME” and exiting These visitors made it all the way to the final step… they didn‟t leave because they are no longer interested! Have FUN! • • • • Get other departments involved to secure buy in “Hey design team, if you could redesign this page what would it look like?” “Hey sales team, what would you change to make the site better?” • Warning, this team is competitive!!!! With A/B Testing the numbers don‟t lie… a variation wins and variations lose
  • 30. First Test Checklist Have I defined my website goals? Have I evaluated where a problem may exist? Have I come up with a solid competing variation(s)? Have I come up with a hypothesis on which variation will win? Have I devised a segment strategy for who sees the test variations? Is one variation outperforming another? Do we have statistical significance? Go with that variation Repeat
  • 31. Now it‟s your turn to become the mad scientist! Q&A
  • 32. Thank You! this webinar prepared by the iSG Digital Services Team at Bridgeline Digital For more information contact iSG at: Or call Seth Rich : (781) 404-5423