Testing in an Automated World

4,088 views
4,036 views

Published on

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,088
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
0
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • Email tests – This story was covered in an interview by eConsultancyThe Obama campaign team carried out A/B tests on 15%-20% of its email list, which included around 15m people in total.Typically the tests involved four different messages that were each split out by three different subject lines. The messages would range from focusing on looming election deadlines to personal pleas from Obama.Some of the tests achieved an additional $2.2m in revenue from a single email by sending the best draft compared to the worst draft. After finding the best subject line and draft Showalter’s team began testing other aspects of the email design, like the unsubscribe language and personalised content.Personalised content included the greeting and the amount to ask for based on each recipient’s previous donation history.The team added ‘drop in’ sentences to target those who had already contributed, such as “thanks for donating last time, please dig deep to donate again.” This type of personalisation significantly raised donation rates. This was kind of revolutionary at the time and overall more than half of the $1bn raised by the campaign came from online donations. This A/B testing probably resulted in $200m in additional revenue.The emails helped to gain donations from more than 4m Americans, recruited tens of thousands of volunteers and publicised events.Takeaways: 1) It can have a huge impact; 2) It can be done over time; and 3)….
  • 3) It’s an opportunity to learn; The more you know about your prospects, consumers and customers, the better your marketing will be and the more competitive your company will be
  • A quick primer, the good & the bad, including some cool things I came across, and a quick dive into the productIntro to A/B TestingWhat can I test?Best practices & Worst practicesMarketo Marketing Automation Features / DemoLanding PagesBatch EmailsTrigger / Engagement Emails
  • Raise your hands if this applies to you
  • You know what happens when you take shortcuts…
  • A quick test – You must participate!These subjects were all tested in an email that was trying to drive registration for the Marketing Sherpa Email Summit. Which do you think won?
  • So where should you be thinking about testing?How many people are testing landing pages today?
  • This is an example of testing your website. There are tons of tools out there, like Optimizely, which is integrated with Marketo.How many people are running AB tests on their website?This example here is actually using Marketo Real-time personalization for AB testing. In RTP, you can test different versions with different audiences, like industries or specific companies.
  • And of course, emails, which is what we’ll mostly talk about today. How many people are running AB tests on their emails today?
  • The #1 goal of testing. But notice I said goal, not RULE. I believe you can learn a lot of small tests. But for the really important tests, this is the right goal.Because if you do this, you can be very confident in the results. Why?
  • Well, we’re going to walk through this math step-by-step because who doesn’t love doing math?Seriously though, AB testing relies on statistics. And that means that when you run a test, you want a result like “I’m 95% confident that these results represent reality”When you send to a large sample, the likelihood of that goes up. But with smaller samples, you may only be able to say “I’m 50% confident…” And then what’s the point?Also, if you try to test multiple things, did you really learn anything? You learned which email will work this one time, but you won’t really know which change caused the different results.
  • So, to that end – if you really want to know whether your results mattered, plug them into a calculator. There’s a link here you can check out, it’s really not hard.
  • Third – NEVER MISS AN OPPORTUNITY TO LEARN SOMETHING.As we’ll see in the product, AB testing is easy enough now to just tack it on to any email you send and learn something new about your audience..Going back to how I started – the more you know about your recipient/visitor/prospect/customer/etc….
  • Last but not least, AB test on things that matter.It’s true that email opens are a good indicator of good subjects, but even that’s going away with mobile pre-headers.To really understand the effectiveness of an email, measure whether the action was completed.
  • Who knows why? But that’s the point of these tests!
  • Just a couple of new things I wanted to include – Amazon tests whether a single big call to action beats deeper links into a site.
  • This company got conversion increases of 25-40%
  • 3) Have a hypothesis and know what you’re testing
  • Product demoBatch testingCustom ConversionTesting in a trigger campaignTesting in an engagement streamCustom Conversion usesFills out formVisits web pageForwarded to a FriendShared on Social Networks
  • ×