Classic A/B test where you decide to show two versions of a specific page. One control version (the current page) and a new one.
However one often forgets that version A was the default version below. So any returning visitors will at least see some inconsistency when they suddenly get to see a B version. This can either lead to more interest from the visitor in that “new” feature or to more barriers out of fear for the new.
Especially with long conversion paths where a visitor comes to your website multiple times, this effect comes into play. This is why in almost all A/B tests, I tend to completely disregard results from the 1st week or so.
Tightly linked to that is the fact that all A/B-testing tools want to see results, and thus they show them as much as they can. Statistical significance is a special subject and one has to be very careful before jumping to conclusions.
Determine sample size before you start OR set up and forget.
Especially in Google Content Experiments, segmentation is almost non-existent. A specific feature might perform completely different on device, different countries, returning vs new visitors, ...
Micro-conversions or semi-conflicting conversions
What if you add social media buttons to a website? Your share will probably go up, but what if it hurts your other (more important) conversions?
So then what do you do when the existing solutions don’t fit your needs any longer? You build your own.
Time check: 6:00, 7:54, 7:25
No redirect needed, which might save a little in page-load time (which also might influence results)
Version C was killed quickly, with version D added.
Time check: 10:42; 12:30, 11:56
Easy implementation, could have some results. However changing colours is something typically linked to running out of ideas to test.
(always be wary of reading conversion test results you’ve seen online. You don’t see the methodology used, and oftentimes the conversion rate that is being looked at is pretty dubious (CTR on a button, rather than end-to-end conversion for example).
The book explains the L.I.F.T. Model for conversion rate optimization. L.I.F.T. – Landing page Influence Function for Tests – contains the key elements that will either contribute to or hinder your customers from taking a desired action. This model is comprised of: 1. Value propositon Evidently this is the most crucial element for any product communication. As your customers say: “What’s in it for me?”. A/B and multivariate testing give you the opportunity to see for yourself which product USP works best for your audience and how you should communicate it. You’ll probably be surprised how difficult it is to define the right USPs and the right way to communicate them, but doing so will help you immensely to understand your customers. It’s essential to discover what’s most important for your customers and then make it easy for them to find what they are looking for. If your customers are looking for a second-hand car, make sure that you’re offering and clearly displaying second-hand cars for sale by using the right USPs. 2. Anxiety While e-commerce is gradually becoming more common, it still raises a lot of questions for quite a few people. Will I get my product on time (or will I even get it at all)? Am I sure this product is really what I want? What happens if the item breaks? How safe are my personal and payment details? There are a whole lot of reasons to delay an action, which usually ends with never seeing that customer again. Let the customers know what will happen with a short message or bullet point. The payment, the delivery, the security system, … Make them feel comfortable, but keep it relevant for your products. Convince customers that they can buy products in a safe and secure way. Sometimes it can also be useful to address product-related concerns with comments like “not tested on animals” (e.g. cosmetics) or “non-irritating” (e.g. cream). 3. Distraction The internet has taken a further toll on our attention spans. For every product your company offers, there are tens, hundreds and, sometimes even, thousands of other companies looking to grab your customer’s attention. So make sure your site not only loads fast (100ms additional delay might cost you 1%), but also doesn’t push too much information on each page. 4. Relevance As mentioned earlier, an endless number of competitors are waiting just around the corner looking to steal that customer from you, so only sell what the customer actually wants. There is no hard sell online, so don’t even try to push the customer in a slightly different direction. 5. Clarity Make sure that everything on your page serves a purpose, and that that purpose matches exactly with the goal your visitor wants to reach. Still need to show multiple call-to-actions? Organize them in a way that there is a clear best choice. Be clear and cohesive in your messages. Only give necessary information and use language that the customers understand. Furthermore, remember that it’s not all about text: the images and the design on the website also contribute to the clarity of your messages. 6. Urgency This is your trump card. Anything limited helps your customer decide right there and then. Simply think of Booking.com saying “only 1 room left” to see how this works, although as with every trick in the playbook, it is important to procede with caution and not try to mislead your customer. AN EXAMPLE FROM BEOBANK While the L.I.F.T.-model is only a guide, we do use the principles in our optimization brainstorming sessions. For Beobank, one of our customers, we are continuously working in improving the ‘Beobank kredietkaart’ pages.
1. Anxiety: An important bullet in the header for Visa users - “Safe internet shopping thanks to Verified by Visa” 2. Distraction: The message and the image appear at the top of the landing page. It grabs user’s attention and contains the information needed to trigger the customers. This has a clean design without any irrelevant images that can distract customers. Clean and clear. 3. Relevance: This landing page is given when customers look for a ‘Visa card’. The bullets in the header and the message blocks below all contain necessary and useful information. But we have kept it as simple as possible. We guard the customers from an overload of irrelevant information. The USPs we use are the most important for our potential customers. 4. Clarity: The benefits of the card are clearly visible - “Receive a reduction of 3% on every purchase online” 5. Urgency: A sentence between the image and the action button - “Ready to start your request?”
PIE = (review PIE model = potential, importance, ease) (three axis = impossible to fit on graph)
Importance (Y) vs Difficulty (X)
Top-left: Easy, important => Do now Right-bottom: Difficult, low impact => Forget Top-right: Difficult, important => strategic decision or split into smaller items Bottom-left: Easy, low impact => Some day
Everything at the top has more impact in relation to its difficulty, so those are the ones to start with.
Expand this with at least another few slides:
Forms Product pages (travel: excursion) FAQ pages (clicks)
Usually the closer to the action taken (purchase, lead), the more important the page becomes. A homepage might be much less important than you think.
I’ve listed some cases to help you get started and be inspired.
Why AirBNB? Because they are testing the way you should be testing: BIG. About user experience, not just small elements.
And I also show you this case to show that AirBNB makes mistakes as well!
If you have conversion flows that take a long while to complete (such as travel or other high-involvement conversions), be very wary of some of the historical effects.
Job ad (organisation) + cross-domain A/B testing
Technical items such as the API used for push notifications²
Imagine if you could test SEO!
Organic Version a is control, version B is updated You clearly see a jump after a few days (after the Google index is updated) of version B After a few weeks I decided to implement version B title to version A as well
The odd thing is that traffic doesn’t seem to have gone up that, but there is huge seasonality in people crocheting stuff. It’s not 100% scientific, but it can work if you make big enough changes. But there are some mistakes I won’t do again in the future:
Taking A/B testing to the next level - Learn from the best:
How does it work
1. Check if your visitor is a recurring visitor and has seen
one of your versions before (cookie)
2. Decide on which A/B/x version to show to your visitor:
if recurring visitor check cookie version
if new visitor % split
3. Set cookie so this visitor will receive same version in
4. Send version number to Google Analytics in custom
dimension (or custom variable)
5. Display version according to our script
6. Set up custom segments in GA to analyze results
1. Check if your visitor is a recurring visitor and has seen one of your versions
2. Decide on which A/B/x version to show to your visitors: new visitor % split,
recurring visitor coookie version
3. Set cookie so this visitor will receive same version in later visits
4. Send version number to Google Analytics in custom dimension (or custom
5. Display version according to our script
6. Set up custom segments in GA to track usage data
A/B testing SEO: correct flow
1. Determine random division of pages from a certain template (database id even (=version A) /
uneven (=version B) could be perfect).
2. Set up a custom dimension (or custom variable) in Google Analytics and send version A and B
3. Let the website continue for at least a few weeks, so you can establish the baseline for your A
and B versions
4. Implement the A/B test by updating title tags for version B only
5. Wait and see if your custom dimension (version B) evolves differently to version A
6. If version B sees increased landings through search engines, copy your best practice to
version A, wait, and see if you end up with an increase in version A as well
• Current A/B testing tools severely limit your options
• Improve your A/B testing significantly by creating your
• Integrate with your central data source for analysis
• You can test pretty much anything: navigation, page
templates, form funnels, social media clicks, SEO
• Keep testing to keep winning!