• Save
Testing in an Automated World
 

Testing in an Automated World

on

  • 3,511 views

 

Statistics

Views

Total Views
3,511
Views on SlideShare
881
Embed Views
2,630

Actions

Likes
5
Downloads
0
Comments
0

9 Embeds 2,630

http://www.marketo.com 2385
http://social2direct.wordpress.com 155
http://uk.marketo.com 22
http://eu.marketo.com 20
https://www.marketo.com 16
http://au.marketo.com 14
http://asia.marketo.com 14
http://mea.marketo.com 2
http://news.google.com 2
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Email tests – This story was covered in an interview by eConsultancyThe Obama campaign team carried out A/B tests on 15%-20% of its email list, which included around 15m people in total.Typically the tests involved four different messages that were each split out by three different subject lines. The messages would range from focusing on looming election deadlines to personal pleas from Obama.Some of the tests achieved an additional $2.2m in revenue from a single email by sending the best draft compared to the worst draft. After finding the best subject line and draft Showalter’s team began testing other aspects of the email design, like the unsubscribe language and personalised content.Personalised content included the greeting and the amount to ask for based on each recipient’s previous donation history.The team added ‘drop in’ sentences to target those who had already contributed, such as “thanks for donating last time, please dig deep to donate again.” This type of personalisation significantly raised donation rates. This was kind of revolutionary at the time and overall more than half of the $1bn raised by the campaign came from online donations. This A/B testing probably resulted in $200m in additional revenue.The emails helped to gain donations from more than 4m Americans, recruited tens of thousands of volunteers and publicised events.Takeaways: 1) It can have a huge impact; 2) It can be done over time; and 3)….
  • 3) It’s an opportunity to learn; The more you know about your prospects, consumers and customers, the better your marketing will be and the more competitive your company will be
  • A quick primer, the good & the bad, including some cool things I came across, and a quick dive into the productIntro to A/B TestingWhat can I test?Best practices & Worst practicesMarketo Marketing Automation Features / DemoLanding PagesBatch EmailsTrigger / Engagement Emails
  • Raise your hands if this applies to you
  • You know what happens when you take shortcuts…
  • A quick test – You must participate!These subjects were all tested in an email that was trying to drive registration for the Marketing Sherpa Email Summit. Which do you think won?
  • So where should you be thinking about testing?How many people are testing landing pages today?
  • This is an example of testing your website. There are tons of tools out there, like Optimizely, which is integrated with Marketo.How many people are running AB tests on their website?This example here is actually using Marketo Real-time personalization for AB testing. In RTP, you can test different versions with different audiences, like industries or specific companies.
  • And of course, emails, which is what we’ll mostly talk about today. How many people are running AB tests on their emails today?
  • The #1 goal of testing. But notice I said goal, not RULE. I believe you can learn a lot of small tests. But for the really important tests, this is the right goal.Because if you do this, you can be very confident in the results. Why?
  • Well, we’re going to walk through this math step-by-step because who doesn’t love doing math?Seriously though, AB testing relies on statistics. And that means that when you run a test, you want a result like “I’m 95% confident that these results represent reality”When you send to a large sample, the likelihood of that goes up. But with smaller samples, you may only be able to say “I’m 50% confident…” And then what’s the point?Also, if you try to test multiple things, did you really learn anything? You learned which email will work this one time, but you won’t really know which change caused the different results.
  • So, to that end – if you really want to know whether your results mattered, plug them into a calculator. There’s a link here you can check out, it’s really not hard.
  • Third – NEVER MISS AN OPPORTUNITY TO LEARN SOMETHING.As we’ll see in the product, AB testing is easy enough now to just tack it on to any email you send and learn something new about your audience..Going back to how I started – the more you know about your recipient/visitor/prospect/customer/etc….
  • Last but not least, AB test on things that matter.It’s true that email opens are a good indicator of good subjects, but even that’s going away with mobile pre-headers.To really understand the effectiveness of an email, measure whether the action was completed.
  • Who knows why? But that’s the point of these tests!
  • Just a couple of new things I wanted to include – Amazon tests whether a single big call to action beats deeper links into a site.
  • This company got conversion increases of 25-40%
  • 3) Have a hypothesis and know what you’re testing
  • Product demoBatch testingCustom ConversionTesting in a trigger campaignTesting in an engagement streamCustom Conversion usesFills out formVisits web pageForwarded to a FriendShared on Social Networks

Testing in an Automated World Testing in an Automated World Presentation Transcript

  • © 2013 Marketo, Inc. Marketo Proprietary and Confidential Matt Zilli Senior Product Marketing Manager Marketo mzilli@marketo.com @mattzilli
  • Page 2 © 2013 Marketo, Inc. Marketo Proprietary and Confidential
  • Page 3 © 2013 Marketo, Inc. Marketo Proprietary and Confidential
  • Agenda 1 2 3 Intro to A/B Testing Best & Worst Practices A/B Testing in Marketo
  • Page 5 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Why aren’t more marketers actively testing?
  • Page 6 © 2013 Marketo, Inc. Marketo Proprietary and Confidential My favorite excuses 1. I have no time for this 2. The results are rarely conclusive 3. It’s too hard! 4. I already know the answer
  • Page 7 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Which Subject Won? Test email subjects from the registration email for the 2014 Marketing Sherpa Email Summit: Your emails don’t work (We’ll tell you why) Middle What happens in Vegas…will improve your emails! Save $300 now. Loser (Open BEFORE Christmas) Email Summit 2014: discount ends Jan 9th Winner Sonia Simone, Copyblogger, Jan 2014
  • Page 8 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Things to test: Landing Pages
  • Page 9 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Things to test: Website
  • Page 10 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Things to test: Emails From name/address Subjects Whole Emails Date/Time
  • Page 11 © 2013 Marketo, Inc. Marketo Proprietary and Confidential 7% Better than B 30% Better than B The loser A. B. C. Jason Fried, signalvnoise.com, Jan 2009 Which headline won?
  • Page 12 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Testing Best Practices
  • Page 13 © 2013 Marketo, Inc. Marketo Proprietary and Confidential #1: Send to a large sample size, test a single variable
  • Page 14 © 2013 Marketo, Inc. Marketo Proprietary and Confidential
  • Page 15 © 2013 Marketo, Inc. Marketo Proprietary and Confidential #2: Check for statistical significance Calculator: www.zettasphere.com/abcalculator
  • Page 16 © 2013 Marketo, Inc. Marketo Proprietary and Confidential #3: Over time, test everything Subject lines, from names/address, content, text, templates, CTAs, creative, button colors, link order….
  • Page 17 © 2013 Marketo, Inc. Marketo Proprietary and Confidential #4: Whenever you can, measure conversions, not opens/clicks
  • 21% Better dmix.ca Blog, 2011
  • Page 19 © 2013 Marketo, Inc. Marketo Proprietary and Confidential New things • Test Deeper CTAs Chris Hexton, MarketingLand
  • Page 20 © 2013 Marketo, Inc. Marketo Proprietary and Confidential New things • Test “Mad Libs” Forms – 25-40% Increases Luke Wroblewski, LukeW Ideation & Design, Feb 2010
  • Page 21 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Things to Avoid: 1) Relying on test results that are insignificant 2) Missing opportunities to test & learn 3) Unstructured tests
  • Page 22 © 2013 Marketo, Inc. Marketo Proprietary and Confidential 12% Higher Click-to-Open Miles Gotcher, Marketo Blog, Oct. 2013
  • Page 23 © 2013 Marketo, Inc. Marketo Proprietary and Confidential How do I do this in Marketo?
  • Page 24 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
  • Page 25 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
  • Page 26 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
  • Page 27 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
  • Page 28 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test Fills out Form Visits web page Forwarded to a Friend Shared on Social Networks
  • Page 29 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
  • Page 30 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Testing in Triggered & Engagement Campaigns
  • Page 31 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Testing in Triggered & Engagement Campaigns
  • Page 32 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Testing in Triggered & Engagement Campaigns
  • Page 33 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Landing Page A/B Test
  • Page 34 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Takeaways 1. Test whenever you can 2. Check for statistical significance 3. Go beyond batch tests 4. Test whole funnel, not just opens/clicks