A/B testing playbook. How to use ClickTale to optimize your A/B testing.

Uploaded on

A/B testing has continually proven to be a superior revenue driver over that of online marketing spend. It enables companies to identify key roadblocks along the conversion funnel, fix flaws in the …

A/B testing has continually proven to be a superior revenue driver over that of online marketing spend. It enables companies to identify key roadblocks along the conversion funnel, fix flaws in the user experience and pinpoint other areas of friction. When executed properly along with both traditional and in-page web analytics, it gives marketers valuable insight that informs optimization and therefore increases revenue.
In this guide, we’ll walk you through each step for using ClickTale to manage and analyze an AB test.
Written by Peter Gorne

More in: Marketing
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. THE A/B TESTING PLAYBOOK How to Use ClickTale to Optimize your A/B Testing
  • 2. 2- THE A/B TESTING PLAYBOOK The importance of A/B testing A/B testing has continually proven to be a superior revenue driver over that of online marketing spend. It enables companies to identify key roadblocks along the conversion funnel, fix flaws in the user experience and pinpoint other areas of friction. When executed properly along with both traditional and in-page web analytics, it gives marketers valuable insight that informs optimization and therefore increases revenue. Once you approach your landing page or website from a testing perspective, the possibilities and combinations are nearly unlimited. Whereas testing may have once been used to simply determine the best size of the “submit” button, companies now commonly test themes such as: Content, Offers, Creative/layout, Pricing, Load time or Functionality. All of these testing options present tremendous opportunity to gain insight, but are a challenge to companies that don’t know where to start. Haphazardly testing elements is a waste of time, money and manpower. All the prior quantitative analysis and customer behavior analysis will hopefully have given us enough insights to decide what and where to test. The real value of any A/B tests comes from the insight gained about visitors’ behavior, and how the copy and design of your site influence it. When you put your energy and resources toward what works, you’ll get the true benefit of your testing program. In this guide, we’ll walk you through each step for using ClickTale to manage and analyze an AB test: > Why you should be using ClickTale for you’re A/B testing? > Preparing for the test > Setting up the test > Analyzing the results. Identifying the winning version > Ideas for tests
  • 3. 3- THE A/B TESTING PLAYBOOK Data driven tests are 7.7 times more effective than UX tests eConsultancy Research Why you should be using ClickTale for your AB testing? One simple reason: context. The problem with traditional AB testing platforms is that their results are very one-dimensional. You choose a single action as your goal and the platform tells you which version caused more of these actions. No micro and macro conversions, no breakdown of revenue generated - no context. With ClickTale, you can add context and value to your testing process. Supplementing traditional testing with in-page analytics will enable you to understand why customers preferred a certain landing page, as opposed to simply tracking which landing page performed better. This is often more important than simply collecting the numbers themselves. Changing the text or visuals on any page will inevitably have an effect on how your customers behaved within the page itself. But is your performance increase due to customers preferring image A over image B? Or simply because the slight page design differences meant that fewer customers scrolled down to see image B in the first place? Testing a product page with long paragraphs of text or simply bullet points will give you clear result as to which encourages more visitors to make a purchase. In-page analytics will then show you if the text itself was superior, or if the length meant that customers didn't even bother to read the text at all. These insights therefore deliver more than just increasing the performance of a single page. Once you can clearly see how your visitors interact with your website as a whole, and what their browsing patterns are, information gleaned from these tests can then legitimately be used to improve the parts of your site that weren't included in the testing process, thus increasing the return on the tests themselves.
  • 4. 4- THE A/B TESTING PLAYBOOK Preparing for the test Deciding what to test The most effective test is a well-researched one. The insight gained from analyzing visitor behavior will help you create a test version that will be much more likely to improve conversion rates. The Conversion Funnel is a good place to start if you haven’t decided which page to test. The easiest way to choose is to build your conversion funnel and choose the page with the lowest conversion rate. The Standard Funnel building method One way to build a funnel is by choosing the most common steps visitors go through. This can be done automatically with ClickTale. The default funnel is loaded with the 3 most popular steps already chosen for you. Simply use the “+Add” buttons to add extra steps to your funnel. The Reverse Funnel building method Another way to build your funnel is to choose the conversion as the first step and work your way backwards. This is primarily used when you don’t know the most common conversion path in your site. To do this, click “Edit” on the first step and choose either your conversion page or conversion Event. Next, copy the URL for the most popular entry page from the “Entry pages” box on the left. Click “+Add” above the first step and paste the entry page URL. Repeat this process until you’ve completed your funnel.
  • 5. 5- THE A/B TESTING PLAYBOOK Determining what to change on your page Once you’ve decided which page to change, you have to figure out what changes to test on your page. This requires the basic ClickTale page analysis. The more thorough you are with your analysis, the easier it will be to decide on changes. Step 1: Behavioral data – Heatmaps, Visitor Recordings Starting with aggregated data (Heatmaps, Form Analytics) is the fastest way to discover areas for improvement. > Mouse Move: check to see which page elements visitors are focusing on and/or ignoring (especially useful if you segment based on converted vs. non-converted). The point is to understand what is influencing conversion decisions and what visitors consider important. > Clicks: check to see where visitors are clicking and what’s causing them to take action. > Scroll Reach: see how far down visitors are scrolling. This is important since you want to make sure your most important information is being seen by your visitors. > Attention: check to see what visitors are reading vs. scanning. This data will give you an idea of how much time to expect visitors to spend looking at your content. Step 2: Statistical data - Form Analytics, Page Console, Events This step is for benchmarking your current macro and micro conversions. > Form Analytics: measure the interaction and completion rates of your form. If few interact with the form, than it’s usually not attractive to visitors or not what they expected. If many interact but few complete, then you most likely have usability issues within your form. > Page Console: a summary of your page activity. It’s very useful to see how long the average visitor spends on your page (amount of time to make your sales pitch) and what were the previous and next pages most visitors went to. > Events: if your ClickTale plan allows for Events, it’s highly recommended to use them. An Event can be added to any part of the page a visitor can interact with, so you can mark conversions, clicks on specific links, visitors arriving from a campaign link, viewing a video, etc.
  • 6. 6- THE A/B TESTING PLAYBOOK Setting up the test There are a few technical aspects to keep in mind when running an AB test even though ClickTale will not be the platform used to redirect traffic and serve up test versions (you will need a 3rd party testing platform for that): > Unique URLs: the easiest way to separate test versions within your ClickTale analytics is by giving each version its own URL (ex: yourdomain.com/version_A and yourdomain.com/version_B) > Events: you can also label pages with events. This is very useful if your testing platform uses the same URL for all test versions. It’s also recommended to add events to any page elements you’d like to track, such as buttons, links, error messages, etc. > Integrations: ClickTale integrates with numerous AB testing platforms, including: Google Content Experiments Optimizely Monetate Adobe Test&Target Visual Website Optimizer Maxymizer Check the Technology Partners section in our website to learn more about these and all other integrations.
  • 7. 7- THE A/B TESTING PLAYBOOK Analyzing the results. Identifying the winning version The fun starts once you have enough data to analyze. We recommend that small sites have at least 250 visitors per Heatmap. Larger sites will obviously need much larger samples to lessen the margin of error. There may be multiple types of conversions in your site. Using the following three conversion measurements will help you get a complete perspective of how your test versions affected your site’s success. We’ll use an example of a pricing page to showcase a test with multiple conversions. Measuring page conversion There are several items we want to measure in this pricing page: > Choosing a plan: visitors can choose from 3 different plans on the pricing page. We can use Link Analytics to measure how many visitors click to choose each plan type. > Requesting a demo: we can use Link Analytics to measure how many visitors click the request a demo button > Starting a chat: just as with the request a demo, we can measure the usage of the chat button with Link Analytics Measuring site conversion Once a visitor clicks on a plan type on the pricing page, they need to complete the signup form on the next page in order to actually convert. To see how many visitors actually converted, we need to measure how many visitors completed the signup form. There are two ways to do this: > Conversion Funnel: see how many visitors went from our pricing page test version to signup page to account welcome page. > Form Analytics: we can segment the signup page based on the previous page of each visitor. By doing this, we can see the completion rate of the form based on which pricing page the visitor came from.
  • 8. 8- THE A/B TESTING PLAYBOOK Measuring the conversion value Even if a visitor signups up for an account, not all conversions are equal. Clicktale.com has several plan tiers from Free to Enterprise with subscription length options. On top of that, not all visitors complete the payment step even when they fill out the signup form for a paid plan. In order to measure the value of each visitor that signs up, we use Events to mark a successful payment (or Free plan signup) as well as an Event for the plan and subscription length purchased. Events can also be configured to mark purchases above or below a certain amount. We can then setup our conversion funnel as shown in the image to measure the revenue generated from each pricing page test version. Measuring behavior to understand why a version won or lost This is where ClickTale’s brand of analytics really comes in handy. Completing an analysis of your test requires understanding why one version beat another. ClickTale’s Heatmaps, Visitor Recordings, and Form Analytics will do just that. What to look for in Heatmaps: > What content did visitors pay attention to? (Mouse Move & Attention) > Did your changes distract visitors from consuming content or encourage them? (Mouse Move) > Which version persuaded visitors to scroll more? (Scroll Reach) > Which version received higher engagement times? (Attention, Page Console) > Did a feature get clicked on more often in one of the test versions? (Click, Link Analytics) > How did behavior change on subsequent pages? (Entire Heatmap Suite) What to look for in Visitor Recordings: > Did visitors make their decision quicker in the test version? (Time spent on page before choosing what to click on) > Did visitors hover over any specific information before clicking or leaving the page? > Did bounced visitors spend any time looking at the content of the page? What to look for in Form Analytics (if testing a form): > Did the percent of visitors that interacted with the form increase? > Did the percent of visitors that submitted the form increase? > Was there a change in time to complete the form?
  • 9. 9- THE A/B TESTING PLAYBOOK Make your Site changes After running your test, it is time to apply what we learnt, change and optimize your Site. But be aware, testing is a learning experience, and sometimes, something it just doesn’t go as expected. You may find yourself in one of the next 2 possible scenarios: Scenario 1. The Site changes don’t perform as expected Why did this happen? Why our changes do not outperform the original version? Well, the scientific method works like this, and you cannot always win. Maybe the experiment was not properly designed and we had not enough traffic and data to take the right decisions. Or we the test preparation and set up was not appropriate. The answer could also be as simple as that our hypotheses were weak and we should reconsider going through the quantitative and qualitative analysis again to rethink our analysis and hypothesis. Scenario 2. The changes perform as expected Though this scenario is the optimal, we have to take into account that though one test version may had clearly overpassed the other version, it does not mean that once implemented we will get the same improvement. Be aware that correlation is not causality. We have just tested a sample of the traffic and statistics are usually more complicated than that! For the same reasons, just because one test over performs the other, it does not mean that is the optimal version. If we test several more versions maybe some of them will perform even better than our winner.
  • 10. 10- THE A/B TESTING PLAYBOOK 10 of the Most Common Testing Ideas: 1. Displaying minimal information vs. comprehensive information on a conversion page 2. Moving the location of call to action buttons (above vs. below the fold) 3. Long vs. short landing pages 4. Headline text messaging – appealing to different personalities/needs 5. Hiding specific information vs. Showing information by default 6. Changing the supporting text to side of the form 7. Testing different image sizes 8. Paragraph text vs. List/Table layout 9. New concept testing (changing many elements of a page at once) 10. Moving more content above the fold vs. Enticing visitors to scroll below the fold Ideas for testing Testing and Optimizing your Website is a never ending process in your quest to achieve Online Success, so don’t get comfortable! It is time to go to step one and start all over again. Here we leave you some of the most common testing ideas:
  • 11. 11- THE A/B TESTING PLAYBOOK About ClickTale ClickTale®, the enterprise leader in Digital Customer Experience, enables businesses to maximize revenues by optimizing the way people experience the Web and Mobile. ClickTale’s comprehensive solution features the industry’s most innovative heatmap technology, playback of individual-user browsing sessions and Customer Experience Consultants dedicated to your success. A subscription-based SaaS solution, ClickTale effortlessly integrates with web analytics, testing tools and any infrastructure to reveal how users really interact with your site. ClickTale creates revenue-winning customer experiences for over 100,000 businesses worldwide, including a large number of Fortune 500 companies. We are committed to making the digital world more enjoyable for people and more lucrative for businesses. For more information, visit www.clicktale.com. US 1-800-807-2117 UK 0800-048-8871 Email: info@clicktale.com © 2014 by ClickTale. All rights reserved. All ClickTale products and services mentioned herein, as well as their respective logos, are trademarked or registered trademarks of ClickTale. All other product and service names mentioned are the trademarks of their respective companies. These materials are subject to change without notice. These materials and the data contained are provided by ClickTale and its customers and partners for informational purposes only, without representation or warranty of any kind, and ClickTale shall not be liable for errors or omissions with respect to the materials, the data or the information, that have been provided by its partners and/or clients for public promotional purposes.