Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evolve Your A/B Testing Skills

5,312 views

Published on

A/B testing is a fundamental skill for growing all aspects of a business. In this talk delivered at MozCon 2016, I lay out a path for evolving from generating a/b testing ideas to hypotheses; from executing tests to automating the process; and from working with a team to encouraging advocates.

Published in: Marketing

Evolve Your A/B Testing Skills

  1. 1. @ c a r a h a r s h m a n U p l e v e l Y o u r A / B T e s t i n g S k i l l s
  2. 2. August 2nd 2016, Central Park, NYC
  3. 3. @ c a r a h a r s h m a n Y o u r A / B T e s t i n g S k i l l sE v o l v e
  4. 4. POKÉMARKETÉRS
  5. 5. A/B TESTING SKILLS EVOLVE YOUR @CaraHarshman #MozCon
  6. 6. 27% 3% 70% 70% OF MARKETERS SAID CONVERSION OPTIMIZATION IS MORE OF A PRIORITY FOR 2016. Source: ConversionXL SAME LESS MORE GREAT NEWS @CaraHarshman #MozCon
  7. 7. 41% 56% 4%56% OF TEAMS WILL ALLOCATE MORE BUDGET TO CONVERSION OPTIMIZATION IN 2016. Source: ConversionXL SAME MORE LESS DECENT NEWS @CaraHarshman #MozCon
  8. 8. BIGGEST OPTIMIZATION CHALLENGES: KNOWLEDGE RESOURCES KNOWING WHAT TO TEST NOT GREAT NEWS @CaraHarshman #MozCon
  9. 9. IDEAS EXECUTION TEAM EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon
  10. 10. IDEAS EXECUTION TEAM EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon HYPOTHESES AUTOMATION ADVOCATES
  11. 11. 1. LEARN RATE = 100% 2. BUSINESS GROWTH, HIGHER REVENUE 3. DELIGHTED CUSTOMERS 4. STRONGER DECISIONS & CULTURE OPTIMAL OUTCOME OF A/B TESTING EVOLUTION @CaraHarshman #MozCon
  12. 12. AN EXAMPLE FROM KYLE RUSH
  13. 13. AN EXAMPLE FROM KYLE RUSH + HILLARY
  14. 14. HYPOTHESIS HILLARY CLINTON CAMPAIGN @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO,
  15. 15. HYPOTHESIS HILLARY CLINTON CAMPAIGN @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO, THEN MORE PEOPLE WILL DONATE,
  16. 16. HYPOTHESIS HILLARY CLINTON CAMPAIGN @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO, THEN MORE PEOPLE WILL DONATE, BECAUSE WE’VE REMOVED THE FRICTION OF PAYMENT.
  17. 17. @CaraHarshman #MozConORIGINAL FLOW
  18. 18. @CaraHarshman #MozConORIGINAL FLOW VARIATION FLOW
  19. 19. RESULTS @CaraHarshman #MozCon CONVERSION LIFT: 239% ORIGINAL CR: 9% VARIATION CR: 31%
  20. 20. VARIATION RESULTS @CaraHarshman #MozCon
  21. 21. IDEAS HYPOTHESES EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon
  22. 22. IDEAS HYPOTHESES “LET’S… ADD THIS MOVE THAT SWAP THIS FOR THAT SHORTEN THIS CHANGE THIS” @CaraHarshman #MozCon
  23. 23. IDEAS HYPOTHESES “LET’S… ADD THIS MOVE THAT SWAP THIS FOR THAT SHORTEN THIS CHANGE THIS” @CaraHarshman #MozCon “IF WE DO THIS THEN THIS WILL HAPPEN BECAUSE OF THIS REASON.”
  24. 24. @CaraHarshman #MozCon X X IDEAS HYPOTHESES
  25. 25. IDEAS EXECUTION TEAM EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon HYPOTHESES AUTOMATION ADVOCATES
  26. 26. @CaraHarshman #MozCon X X ANALYTICS VOICE OF THE CUSTOMER IDEAS HYPOTHESES
  27. 27. GREAT A/B TESTS HYPOTHESES ARE BORN OUT OF REASON & DATA, NOT WHITEBOARD SESSIONS @CaraHarshman #MozCon
  28. 28. ANALYTICS @CaraHarshman #MozCon QUANTITATIVE / QUALITATIVE INFORMATION ABOUT YOUR WEBSITE/APP
  29. 29. TOP ANALYTICS SOURCES @CaraHarshman #MozCon 1. FUNNEL REPORT (GA) 2. HEAT MAP ON HOME & HIGH INTENT PAGES 3. FORM ERROR SUBMISSION REPORT (GA) 4. EXIT / PAGE BOUNCE REPORT (GA) 5. HIGH TRAFFIC / LOW CONVERSION LANDING PAGES
  30. 30. VOICE OF THE CUSTOMER @CaraHarshman #MozCon EMPATHY BASED ANALYSIS, UNDERSTAND THE NEEDS OF THE END USER AS A HUMAN
  31. 31. VOICE OF THE CUSTOMER LISTENING CHANNELS @CaraHarshman #MozCon 1. CUSTOMER SUCCESS TEAM 2. RECORDED USER TESTING SESSIONS 3. COFFEE SHOPS 4. ON-PAGE SURVEYS 5. EMAIL SURVEYS
  32. 32. VOICE OF THE “HUMAN” @CaraHarshman #MozCon 1. LIFT MODEL BY WIDER FUNNEL 2. CONVERSION EQUATION FROM OLI GARDNER 3. PSYCHOLOGY 4. DESIGN PRINCIPLES 5. NEUROMARKETING
  33. 33. MVMT WATCHES A HYPOTHESIS CASE STUDY
  34. 34. HYPOTHESIS MVMT WATCHES @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE,
  35. 35. HYPOTHESIS MVMT WATCHES @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE, THEN SALES WILL INCREASE
  36. 36. HYPOTHESIS MVMT WATCHES @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE, THEN SALES WILL INCREASE BECAUSE THIS PLACEMENT MIMICS THE IN-STORE BUYING EXPERIENCE.
  37. 37. VARIATION MVMT WATCHES @CaraHarshman #MozCon@CaraHarshman #MozCon
  38. 38. RESULTS @CaraHarshman #MozCon CONVERSION LIFT DESKTOP: 2.2% MOBILE: 5.5%
  39. 39. @CaraHarshman #MozCon
  40. 40. A/B TEST HACKATHON @CaraHarshman #MozCon 1. GIVE PEOPLE ACCESS TO DATA 2. INVITE CROSS FUNCTIONAL TEAMS 3. GIVE PRIZES 4. FOLLOW UP
  41. 41. TESTING HACKATHON POSTER DOWNLOAD OPTIMIZELY.COM/TESTING-TOOLKIT/
  42. 42. @CaraHarshman #MozCon X X ANALYTICS VOICE OF THE CUSTOMER IDEAS HYPOTHESES
  43. 43. EXECUTION AUTOMATION EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon GET SHIT DONE
  44. 44. EXECUTION AUTOMATION @CaraHarshman #MozCon GET SHIT DONEGET SHIT DONE
  45. 45. EXECUTION AUTOMATION @CaraHarshman #MozCon GET SHIT DONE MAKE A PLAN SO WE CAN SCALE
  46. 46. @CaraHarshman #MozCon GET SHIT DONE X X PRIORITIZATION STANDARDIZATION EXECUTION AUTOMATION
  47. 47. IF YOU WANT TO SCALE, THEN YOU HAVE TO TAKE OFF YOUR GET-SHIT-DONE-HAT Image source: http://teeeshop.com/products/get-shit-done-bucket-hat
  48. 48. PRIORITIZATION @CaraHarshman #MozCon AN EMOTION-FREE SYSTEM TO DECIDE WHAT TO TEST
  49. 49. PRIORITIZATION CHART: EASE VS. IMPACT @CaraHarshman #MozCon IF YOU CAN MAKE TIME DEPRIORITIZE NEEDS A VERY SOLID HYPOTHESIS DO IT! EASE IMPACT
  50. 50. PRIORITIZATION SCORE CARD @CaraHarshman #MozCon RULE 1 point 0 points page part of funnel outside funnel audience 100% of visitors specific audience success metric primary goal secondary goal A RULE & POINT BASED SYSTEM WHERE EVERY HYPOTHESIS GETS A SCORE
  51. 51. “IF YOU COME TO ME WITH AN IDEA AND IT’S NOT LIVE IN 2 WEEKS, IT’S NOT BECAUSE IT’S A BAD IDEA – IT’S BECAUSE I HAVE BETTER THINGS TO TEST.” -Pauline, PM at Hotwire
  52. 52. PRIORITIZATION WORKSHEETS DOWNLOAD OPTIMIZELY.COM/TESTING-TOOLKIT/
  53. 53. STANDARDIZATION @CaraHarshman #MozCon A BROADLY AGREED-UPON SET OF PRIORITIES, METRICS, DEFINITIONS
  54. 54. 3 PIECES TO STANDARDIZE @CaraHarshman #MozCon 1. MAIN SUCCESS METRICS 2. STOPPING CRITERIA FOR TESTS 3. IMPORTANT AUDIENCES
  55. 55. STANDARDIZE SUCCESS METRICS @CaraHarshman #MozCon WHAT ARE THE ACTIONS YOU ARE TRYING TO MOVE? COMPANY GOALS TEAM GOALS TESTING GOALS TEST HYPOTHESIS GOAL
  56. 56. STANDARDIZE STOPPING CRITERIA @CaraHarshman #MozCon HOW LONG WILL A TEST RUN BEFORE YOU STOP IT? STOP
  57. 57. STANDARDIZE YOUR AUDIENCES @CaraHarshman #MozCon WHICH UNIQUE TRAFFIC SEGMENTS DESERVE MORE ATTENTION? New Google Mobile Desktop Returning Referral
  58. 58. BROOKS RUNNING A STANDARDIZATION CASE STUDY
  59. 59. HYPOTHESIS BROOKS RUNNING @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES,
  60. 60. HYPOTHESIS BROOKS RUNNING @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES, THEN THEY WILL BE LESS LIKELY TO RETURN
  61. 61. HYPOTHESIS BROOKS RUNNING @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES, THEN THEY WILL BE LESS LIKELY TO RETURN BECAUSE THEY DECIDED ON A SIZE.
  62. 62. @CaraHarshman #MozCon EXPERIENCE FOR INDECISIVE SHOPPERS
  63. 63. RESULTS @CaraHarshman #MozCon RETURN RATE FOR THIS AUDIENCE: -80% CONVERSION RATE: +2%
  64. 64. @CaraHarshman #MozCon GET SHIT DONE X X PRIORITIZATION STANDARDIZATION EXECUTION AUTOMATION
  65. 65. TEAM ADVOCATES EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon
  66. 66. TEAM ADVOCATES @CaraHarshman #MozCon PEOPLE WHO DO WORK
  67. 67. TEAM ADVOCATES @CaraHarshman #MozCon PEOPLE WHO DO WORK PEOPLE WHO PROMOTE YOUR WORK
  68. 68. @CaraHarshman #MozCon X X COMMUNICATION DIVERSE SKILLS TEAM ADVOCATES
  69. 69. 30% 29% 20% 21% WHO DOES OPTIMIZATION AT YOUR ORGANIZATION? Source: ConversionXL (PRETTY) GOOD NEWS @CaraHarshman #MozCon NOBODY A TEAM OF PEOPLE ONE PERSON PEOPLE ACROSS TEAMS
  70. 70. TOP 5 SKILL SETS TO HAVE @CaraHarshman #MozCon 1. PROJECT MANAGER 2. DEVELOPER 3. DESIGNER 4. ANALYST 5. FEARLESSNESS
  71. 71. 2 TEAM EXAMPLES @CaraHarshman #MozCon JOSEPHINE AT HOTWIRE DESIGN DEVELOPER CUSTOMER SUCCESS PRODUCT OWNER, HOTELS PRODUCT OWNER, FLIGHTS CRAIG HEAD OF GROWTH JOSEPHINE PRODUCT MANAGER FOR TESTING CRAIG AT EARNEST
  72. 72. COMMUNICATION CHANNELS @CaraHarshman #MozCon 1. OPT-IN EMAIL NEWSLETTER 2. WEEKLY MEETINGS FOR CORE CREW 3. INTERNAL WIKI / WORKFLOW TOOL 4. LUNCH N’ LEARNS 5. ALL HANDS MEETINGS
  73. 73. RESULTS SHARING TEMPLATES + DECKS DOWNLOAD OPTIMIZELY.COM/TESTING-TOOLKIT/
  74. 74. IDEAS HYPOTHESES EXECUTION AUTOMATION TEAM ADVOCATES EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon
  75. 75. 1. LEARN RATE = 100% 2. BUSINESS GROWTH, HIGHER REVENUE 3. DELIGHTED CUSTOMERS 4. STRONGER DECISIONS & CULTURE OPTIMAL OUTCOME OF A/B TESTING EVOLUTION @CaraHarshman #MozCon
  76. 76. C A R A H A R S H M A N c a r a h a r s h m a n . m e @ c a r a h a r s h m a n

×