Deep Dive: Testing in an Automated World


Published on

Marketers have long struggled with email testing. Deciding when, how, and what to test can get overwhelming very quickly. When you're running automated campaigns on top of that, all but the most dedicated marketers among us tend to throw testing out the window.

Don't despair! Marketo's Senior Product Marketing Manager, Matt Zilli, has the answer! In this can't-miss webinar, he will dive into:

- Email testing best practices
- The latest in testing functionality
- Much more!

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Email tests – This story was covered in an interview by eConsultancy

    The Obama campaign team carried out A/B tests on 15%-20% of its email list, which included around 15m people in total.

    Typically the tests involved four different messages that were each split out by three different subject lines. The messages would range from focusing on looming election deadlines to personal pleas from Obama.

    Some of the tests achieved an additional $2.2m in revenue from a single email by sending the best draft compared to the worst draft. 

    After finding the best subject line and draft Showalter’s team began testing other aspects of the email design, like the unsubscribe language and personalised content.

    Personalised content included the greeting and the amount to ask for based on each recipient’s previous donation history.

    The team added ‘drop in’ sentences to target those who had already contributed, such as “thanks for donating last time, please dig deep to donate again.” This type of personalisation significantly raised donation rates. 

    This was kind of revolutionary at the time and overall more than half of the $1bn raised by the campaign came from online donations. This A/B testing probably resulted in $200m in additional revenue.

    The emails helped to gain donations from more than 4m Americans, recruited tens of thousands of volunteers and publicised events.

    Takeaways: 1) It can have a huge impact; 2) It can be done over time; and 3)….
  • 3) It’s an opportunity to learn; The more you know about your prospects, consumers and customers, the better your marketing will be and the more competitive your company will be
  • A quick primer, the good & the bad, including some cool things I came across, and a quick dive into the product

    Intro to A/B Testing
    What can I test?

    Best practices & Worst practices

    Marketo Marketing Automation Features / Demo
    Landing Pages
    Batch Emails
    Trigger / Engagement Emails
  • Raise your hands if this applies to you
  • You know what happens when you take shortcuts…
  • A quick test – You must participate!

    These subjects were all tested in an email that was trying to drive registration for the Marketing Sherpa Email Summit. Which do you think won?
  • A quick test – You must participate!

    These subjects were all tested in an email that was trying to drive registration for the Marketing Sherpa Email Summit. Which do you think won?
  • So where should you be thinking about testing?

    How many people are testing landing pages today?
  • This is an example of testing your website. There are tons of tools out there, like Optimizely, which is integrated with Marketo.

    How many people are running AB tests on their website?

    This example here is actually using Marketo Real-time personalization for AB testing. In RTP, you can test different versions with different audiences, like industries or specific companies.
  • And of course, emails, which is what we’ll mostly talk about today. How many people are running AB tests on their emails today?
  • The #1 goal of testing. But notice I said goal, not RULE. I believe you can learn a lot of small tests. But for the really important tests, this is the right goal.

    Because if you do this, you can be very confident in the results. Why?
  • Well, we’re going to walk through this math step-by-step because who doesn’t love doing math?

    Seriously though, AB testing relies on statistics. And that means that when you run a test, you want a result like “I’m 95% confident that these results represent reality”

    When you send to a large sample, the likelihood of that goes up. But with smaller samples, you may only be able to say “I’m 50% confident…” And then what’s the point?

    Also, if you try to test multiple things, did you really learn anything? You learned which email will work this one time, but you won’t really know which change caused the different results.
  • So, to that end – if you really want to know whether your results mattered, plug them into a calculator. There’s a link here you can check out, it’s really not hard.

    As we’ll see in the product, AB testing is easy enough now to just tack it on to any email you send and learn something new about your audience..

    Going back to how I started – the more you know about your recipient/visitor/prospect/customer/etc….
  • Last but not least, AB test on things that matter.

    It’s true that email opens are a good indicator of good subjects, but even that’s going away with mobile pre-headers.

    To really understand the effectiveness of an email, measure whether the action was completed.

  • Who knows why? But that’s the point of these tests!
  • Who knows why? But that’s the point of these tests!
  • Just a couple of new things I wanted to include – Amazon tests whether a single big call to action beats deeper links into a site.
  • This company got conversion increases of 25-40%
  • 3) Have a hypothesis and know what you’re testing
  • Product demo
    Batch testing
    Custom Conversion
    Testing in a trigger campaign
    Testing in an engagement stream

    Custom Conversion uses
    Fills out form
    Visits web page
    Forwarded to a Friend
    Shared on Social Networks
  • ×