Best Practice Webinar: Effective A/B Tests [Oct. 18, 2012]
Testing for SuccessHow to Effectively Utilize A/B Split Testing
Fishbowl Best Practices WebinarHosted by:Drew HebbleSenior Marketing Specialist
AGENDA• The Basics• Getting Started• Improving Open Rates• Improving ClickThrough Rates• After the A/B Split Test• Q&A
THE BASICS:WHAT IS AN A/B SPLIT TEST & HOW DOES IT WORK
What is an A/B Split Test?A simple method for comparing the performance of two different versions of an e-mail campaign
How Does an A/B Test Split Work?• A small subset of the total member list is selected as a test group and divided into two equal parts.• Each half of the test group receives a different version of the same message and a winner is identifiedThe higher-performing message is blasted to the remainder of the list.
How Will A/B Split Testing Help my eClub?Understanding the behavior of your e-club members will help youconstruct a messaging strategy that better engages them Engaged members drive: – Higher open rates – Higher click-through rates – Lower unsubscribe rates THE BETTER YOUR PROGRAM PERFORMS AGAINST THESE KEY METRICS, THE GREATER THE SALES IMPACT!
GETTING STARTED:CHOOSING THE APPROPRIATE A/B SPLIT TEST
Getting Started: What Should You Test? Each A/B split test should isolate a specific message component that has the potential to provide actionable results Common test variables include: Subject lines Creative design Call to action language Call to action placement Call to action style Actions based on Send times (time of day or week) findings From lines
Getting Started: What Should I Measure?Open Rate• Key Test Variables – Subject line Open CTR – Time of day & day of weekClick-through Rate (CTR)• Key Test Variables – Call to action What’s Most • Language, Common • Style • Placement
Getting Started: Guidelines for a Clean & Effective Test1. Restrict each test to 1 variable A comparison of two subject lines sent at different times of day makes it difficult to directly correlate results with to one of the variables2. Make sure that your results can be measured Comparing two non-clickable calls to action would not provide an action to measure3. Construct a fair comparison “Free appetizer today!” vs. “Your October Restaurant Newsletter” are too different to provide a fair comparison.
IMPROVING OPEN RATES:A/B SPLIT TESTING OF SUBJECT LINES
Subject Line Testing: Good Examples• Offer “teaser” vs. Specific offer Enjoy this Special Birthday Gift! Enjoy your Free Entrée Birthday Gift!• Personalization Drew, enjoy a free dessert on us! Enjoy a free dessert on us!• Subject line length We’ve got it all! We’ve got it all! Good food, music, giveaways and more!
Subject Line Testing: Good Examples• Symbols Rainy day special! Rainy day special!• Placement of offer copy Free Appetizer for you and a Friend! You and a Friend can enjoy a Free Appetizer! Adjust testing types, based on your branding, assumptions and historical data
Subject Line Testing: Is It a “TRUE” Test? Have a Free Appetizer on us! vs. Drew, we’d like to offer you a Free Appetizer – today only! You’re going to love this great deal! vs. CAUTION… UNFAIR A Free Entrée for you and a friend! COMPARISON AHEAD
Subject Line Testing: “I’ve sent my test, now what?”1. Review key performance metrics for each version of the message (opens, click-throughs, and unsubscribes)2. Send the top performing subject line to remaining members3. Monitor winning subject line performance?4. Analyze the results and use them to improve on next campaign? The results of each test should add one more piece to the puzzle
What Does Winning Performance Look Like….. 25% 20% IMPORTANT! 15% Consider ALL messagingmetrics, NOT just open rate 10% 5% 0% Open Rate ClickThrough Rate Unsubscribe Rate Subject Line A: Enjoy this Special 19% 1.20% 0.12% Birthday Gift! Subject Line B: Enjoy your Free Entrée 23% 1.40% 0.13% Birthday Gift!
IMPROVING CLICKTHROUGH RATES:A/B TESTING OF CALL TO ACTION
Call to Action Testing: Language• Question vs. statement• Click Here vs. (an implied clickthrough icon) Want a Free Click Here for your Appetizer? Free Appetizer
Call to Action Testing: Placement• Left vs. right• Top vs. bottom• Inside an image vs. stand alone Happy Birthday Happy on Us! Birthday on Us! Click Here for a free brownie during the Click Here for a week of free brownie your birthday. during the week of your birthday.
Call to Action Testing: Visual Style• Clickable body copy vs. button• Animation vs. static image• Different color copy/button To help you through To help you through this rainy this rainy weather weather, Click here to enjoy a free Click Here for your appetizer with any free appetizer with entrée purchase. any entrée purchase.
Call to Action Testing: Is It A “TRUE” Test • Different images on each version • One has a clickable call to action, the other has no measurable variableAVOID UNFAIRCOMPARISONS Click Here for your free appetizer with any entrée purchase.
Improving CTR: “I’ve sent my test, now what?”1. Review click-through performance metrics for each version of the message2. Send top performing call to action to remaining members3. Monitor winning call to action performance across the remaining audience?4. Analyze the results and use them to improve on next campaign? Want a Free Click Here for your Free Appetizer Appetizer? Version A (2.8% CTR) Version B (1.7% CTR)
AFTER THE A/B TEST:ANALYZING RESULTS & APPLYING LEARNING’S
There Was No Clear Winner…What Should I Do?• Both versions performed below your average? – Negative results can be as useful as positive ones. Learn from what your guests don’t respond to, and try a different strategy on your next campaign.• Click-through results don’t lean one way or the other? – Review your testing method to ensure that there the difference between the two message versions is significant. A “Click here to RSVP” vs. “Click below to RSVP” call to action may not yield a different result
There Was No Clear Winner…What Should I Do?One version had a much higher open rate, while the other version had a far superior clickthrough rate? It’s important to consider the amount of variation between metrics. Depending on the focus of your message, it’s sometimes better to Subject line test results sacrifice one slightly lower metric for another that is much greater. A) 23% open rate & 1.1% CTR B) 21% open rate & 3.3% CTR
Acting on Your Results • Shorter subject line outperformed a longer subject line? • Red call to action yielded a higher click-through than a blue one?Make Assumptions based on learning and test on future campaigns Are you able to replicate the results If YES – You’re on the right track to better understanding your audience. If NO – Was there a difference on this test that may have swayed results? Refine your methodology and continue to test until results are consistent.
Three Key A/B Testing Takeaways1. ALWAYS test variable at a time2. ALWAYS review the performance of your entire mailing, NOT just the A/B test3. ALWAYS apply your learnings to future campaigns4. NEVER stop testing and refining
Upcoming Fishbowl WebinarsOctober 18th Optimizing your website to attract new e-club membersNovember 15th Messaging Best Practices
Resources For more information on Fishbowl’s Creative and strategic Professional Services offerings:Enterprise Clients: Please contact your Account DirectorLocal Clients: Please contact your Sales Representative Technical questions on how to do A/B subject line tests? Call our Support line 800.883.1984 or email email@example.com Sign up for a webinar: http://www.fishbowl.com/feature/webinar_enterprise_series
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.