• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Patrick McKenzie Opticon 2014: Advanced A/B Testing
 

Patrick McKenzie Opticon 2014: Advanced A/B Testing

on

  • 2,647 views

A/B Testing Beyond Headlines and Button Colors -- ideas for tests (particularly for B2B SaaS), common pitfalls in organizations, and how to overcome them.

A/B Testing Beyond Headlines and Button Colors -- ideas for tests (particularly for B2B SaaS), common pitfalls in organizations, and how to overcome them.

Statistics

Views

Total Views
2,647
Views on SlideShare
2,161
Embed Views
486

Actions

Likes
7
Downloads
21
Comments
0

4 Embeds 486

http://inbound.org 361
https://twitter.com 121
https://www.rebelmouse.com 3
http://www.slideee.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Patrick McKenzie Opticon 2014: Advanced A/B Testing Patrick McKenzie Opticon 2014: Advanced A/B Testing Presentation Transcript

    • Conversion Optimization: The World Beyond Headlines and Button Color Patrick McKenzie Founder, Kalzumeus Software
    • Low hanging fruit tastes great, but the rest is worth the work, too.
    • Test Idea #1 Quotas / Tier Levels / etc
    • @user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
    • What Did We Learn From That Test? • ~60% increase in sales going from 15 -> 8 cards • Synergistic effects can occasionally actually happen. • You don’t know what you don’t know until you test for it.
    • Key Points With Testing Quotas • Grandfather in existing users. • Set quota levels intelligently. • Be ready to tolerate small uptick in support load. • Your product is still too cheap.
    • Test Idea #2 Payment Methods / Checkout Flow
    • Threw This Together In An Hour (Oops)
    • Rethought Design & UX
    • What We Learned From This Test • A single A/B test can help your bottom line more than a product launch. • Dealing with synergistic effects in testing can occasionally be maddening. • The data is never wrong, but sometimes the data isn’t telling you what you think it is.
    • Key Points With Testing Checkout Flow • Make absolutely sure you don’t break this! • Have UX-focused team (product / marketing / etc) go over every step of checkout funnel with zero tolerance for bad UX. • Checkout flows go “stale” at many successful companies. Schedule a deep dive on them twice a year, at least. • Upsells and cross selling are classics for a reason!
    • Test Idea #3 Offline Business Processes
    • Testing degree of sales which maximizes conversion of leads
    • What We Learned From This Test • Important to communicate with sales/CS. • Lead to resource-intensive but hugely successful lead-scoring efforts. • Justified very expensive interventions like e.g. onboarding help for everybody.
    • Key Points With Testing Offline Processes • Your software/workflow has to close the loop back to the testing software. This can be tricky. • Virtually no CRM or A/B testing software accurately models enterprise decisionmaking process. You might be forced to optimize for proxy conversions. • Volumes required for statistical significance are really punishing, so sometimes we “proxy and pray.”
    • Test Idea #4 Product On-boarding / FRE
    • What Engineers Will Ship If You Let Them
    • Key Goals of Onboarding Process • Get user to a concrete success as soon as possible. • For more complex sales, lay groundwork for future success by e.g. inviting in team or getting permission to follow-up. • For B2C, especially in social, you live and die on virality during the onboarding experience.
    • In-app Tour Introducing Core Use
    • Cheat Code for B2B SaaS Sales
    • What We Learned From This Test • Users who start onboarding tour 25% to 100%+ more likely to purchase software • Users sometimes have visceral reaction to elements of the onboarding experience. • Engineers prefer in-app onboarding but it is expensive as heck to build and does not always justify itself versus e.g. a Youtube video.
    • Key Points With Testing Offline Processes • Your software/workflow has to close the loop back to the testing software. This can be tricky. • Virtually no CRM or A/B testing software accurately models enterprise decisionmaking process. You might be forced to optimize for proxy conversions. • Volumes required for statistical significance are really punishing, so sometimes we “proxy and pray.”
    • Test Idea #5 Core Product Interactions
    • Too Complicated
    • 80% of Users Use 20% of Features
    • Try Radically Simplifying
    • Improved Presentation and UX Of Core Workflow
    • What We Learned From This Test • Just hiding the advanced features increased task success by 10%, which directly improved sales • Further jump of 15% when we did the redesign – Absolutely no increase in sales… – … but we saw in the funnel that this was solely due to less people hitting the trial limits … – … which we were able to profitably exploit with the quota limit test described earlier
    • Key Points With Testing Core Interactions • Test the entire funnel. Sometimes task success doesn’t (unfortunately) help bottom line metrics. • A good understanding of where people spend their time/energy is vital – test the 90% interaction, not the 2%.
    • ROI on ~$500 A Month Testing Budget 0.00% 0.50% 1.00% 1.50% 2.00% 2.50% 3.00% 3.50% February,2010 March,2010 April,2010 May,2010 June,2010 July,2010 August,2010 September,2010 October,2010 November,2010 December,2010 January,2011 February,2011 March,2011 April,2011 May,2011 June,2011 July,2011 August,2011 September,2011 October,2011 November,2011 December,2011 January,2012 February,2012 March,2012 April,2012 May,2012 June,2012 July,2012 August,2012 September,2012 October,2012 November,2012 December,2012 January,2013 February,2013 March,2013 April,2013 May,2013 June,2013 July,2013 August,2013 September,2013 October,2013 November,2013 December,2013 January,2014 February,2014
    • Let’s Talk Testing Challenges
    • A/B Testing Across System Boundaries
    • Not A Hard Problem Our Marketing Site A/B
    • Gets A Wee Bit Trickier With 2 Sites Our Marketing Site A/B The Web Application
    • Total Freaking Madness Our Marketing Site A/B The Web Application iPhone app Offline processes Android app Partner site Content minisite Offsite checkout Japanese subsidiary
    • Sanity Restored Consumer Applications Identity & A/B Manager
    • Lessons Learned With This • Think about strategy for transitioning people between systems – easier to build into design earlier than later • Cookies with unique identities are a stopgap, not a solution. You need to understand the sales process. • This is a good opportunity to standardize on an A/B testing tool and workflow… or you’ll hate your life • This is a “high class problem” which can be deferred until $50M+ in revenue, but when you solve it, it is going to cost serious time/money.
    • Increasing A/B Testing Adoption In Engineering and Marketing Teams
    • Single Most Important Lesson: One Line of Code @user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
    • Other Lessons Learned • Organization should learn rather than guess better. – Best investment ever: organization-wide wiki. – Next best: weekly “What we tried this week” email. • Engineers love A/B testing when you describe it like a puzzle or a video game, with bragging rights. • Marketers love A/B testing when it’s positioned as a way to increase their status in the organization rather than a replacement for their talents.
    • Things To Never Be Without • CS should be able to force-override a user’s participation in an A/B test (and, relatedly, “see what users is seeing” is great if you can get it). • A/B tests deeper than surface level need unit/etc tests. Make sure your framework supports it. • Test switcher UI in development/staging is a huge win. • URL query parameter to flip tests a huge win.
    • Thanks for Listening! • I love talking about this and you cannot possibly waste my time: @patio11 or patrick@kalzumeus.com • Free email mini-course on CRO in software companies: http://bit.ly/patrick-optimizely • Liked this presentation? I deliver something like this twice a month, free, to my email list. http://training.kalzumeus.com to sign up