Evolve Your A/B Testing Skills

3,782 views

Published on

A/B testing is a fundamental skill for growing all aspects of a business. In this talk delivered at MozCon 2016, I lay out a path for evolving from generating a/b testing ideas to hypotheses; from executing tests to automating the process; and from working with a team to encouraging advocates.

Published in: Marketing
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,782
On SlideShare
0
From Embeds
0
Number of Embeds
3,398
Actions
Shares
0
Downloads
9
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Evolve Your A/B Testing Skills

  1. 1. @ c a r a h a r s h m a n U p l e v e l Y o u r A / B T e s t i n g S k i l l s
  2. 2. August 2nd 2016, Central Park, NYC
  3. 3. @ c a r a h a r s h m a n Y o u r A / B T e s t i n g S k i l l sE v o l v e
  4. 4. POKÉMARKETÉRS
  5. 5. A/B TESTING SKILLS EVOLVE YOUR @CaraHarshman #MozCon
  6. 6. 27% 3% 70% 70% OF MARKETERS SAID CONVERSION OPTIMIZATION IS MORE OF A PRIORITY FOR 2016. Source: ConversionXL SAME LESS MORE GREAT NEWS @CaraHarshman #MozCon
  7. 7. 41% 56% 4%56% OF TEAMS WILL ALLOCATE MORE BUDGET TO CONVERSION OPTIMIZATION IN 2016. Source: ConversionXL SAME MORE LESS DECENT NEWS @CaraHarshman #MozCon
  8. 8. BIGGEST OPTIMIZATION CHALLENGES: KNOWLEDGE RESOURCES KNOWING WHAT TO TEST NOT GREAT NEWS @CaraHarshman #MozCon
  9. 9. IDEAS EXECUTION TEAM EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon
  10. 10. IDEAS EXECUTION TEAM EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon HYPOTHESES AUTOMATION ADVOCATES
  11. 11. 1. LEARN RATE = 100% 2. BUSINESS GROWTH, HIGHER REVENUE 3. DELIGHTED CUSTOMERS 4. STRONGER DECISIONS & CULTURE OPTIMAL OUTCOME OF A/B TESTING EVOLUTION @CaraHarshman #MozCon
  12. 12. AN EXAMPLE FROM KYLE RUSH
  13. 13. AN EXAMPLE FROM KYLE RUSH + HILLARY
  14. 14. HYPOTHESIS HILLARY CLINTON CAMPAIGN @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO,
  15. 15. HYPOTHESIS HILLARY CLINTON CAMPAIGN @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO, THEN MORE PEOPLE WILL DONATE,
  16. 16. HYPOTHESIS HILLARY CLINTON CAMPAIGN @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE MAKE IT AS EASY AS POSSIBLE TO SAVE YOUR CREDIT CARD INFO, THEN MORE PEOPLE WILL DONATE, BECAUSE WE’VE REMOVED THE FRICTION OF PAYMENT.
  17. 17. @CaraHarshman #MozConORIGINAL FLOW
  18. 18. @CaraHarshman #MozConORIGINAL FLOW VARIATION FLOW
  19. 19. RESULTS @CaraHarshman #MozCon CONVERSION LIFT: 239% ORIGINAL CR: 9% VARIATION CR: 31%
  20. 20. VARIATION RESULTS @CaraHarshman #MozCon
  21. 21. IDEAS HYPOTHESES EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon
  22. 22. IDEAS HYPOTHESES “LET’S… ADD THIS MOVE THAT SWAP THIS FOR THAT SHORTEN THIS CHANGE THIS” @CaraHarshman #MozCon
  23. 23. IDEAS HYPOTHESES “LET’S… ADD THIS MOVE THAT SWAP THIS FOR THAT SHORTEN THIS CHANGE THIS” @CaraHarshman #MozCon “IF WE DO THIS THEN THIS WILL HAPPEN BECAUSE OF THIS REASON.”
  24. 24. @CaraHarshman #MozCon X X IDEAS HYPOTHESES
  25. 25. IDEAS EXECUTION TEAM EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon HYPOTHESES AUTOMATION ADVOCATES
  26. 26. @CaraHarshman #MozCon X X ANALYTICS VOICE OF THE CUSTOMER IDEAS HYPOTHESES
  27. 27. GREAT A/B TESTS HYPOTHESES ARE BORN OUT OF REASON & DATA, NOT WHITEBOARD SESSIONS @CaraHarshman #MozCon
  28. 28. ANALYTICS @CaraHarshman #MozCon QUANTITATIVE / QUALITATIVE INFORMATION ABOUT YOUR WEBSITE/APP
  29. 29. TOP ANALYTICS SOURCES @CaraHarshman #MozCon 1. FUNNEL REPORT (GA) 2. HEAT MAP ON HOME & HIGH INTENT PAGES 3. FORM ERROR SUBMISSION REPORT (GA) 4. EXIT / PAGE BOUNCE REPORT (GA) 5. HIGH TRAFFIC / LOW CONVERSION LANDING PAGES
  30. 30. VOICE OF THE CUSTOMER @CaraHarshman #MozCon EMPATHY BASED ANALYSIS, UNDERSTAND THE NEEDS OF THE END USER AS A HUMAN
  31. 31. VOICE OF THE CUSTOMER LISTENING CHANNELS @CaraHarshman #MozCon 1. CUSTOMER SUCCESS TEAM 2. RECORDED USER TESTING SESSIONS 3. COFFEE SHOPS 4. ON-PAGE SURVEYS 5. EMAIL SURVEYS
  32. 32. VOICE OF THE “HUMAN” @CaraHarshman #MozCon 1. LIFT MODEL BY WIDER FUNNEL 2. CONVERSION EQUATION FROM OLI GARDNER 3. PSYCHOLOGY 4. DESIGN PRINCIPLES 5. NEUROMARKETING
  33. 33. MVMT WATCHES A HYPOTHESIS CASE STUDY
  34. 34. HYPOTHESIS MVMT WATCHES @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE,
  35. 35. HYPOTHESIS MVMT WATCHES @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE, THEN SALES WILL INCREASE
  36. 36. HYPOTHESIS MVMT WATCHES @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE ADD THE NEW PRODUCT UP-SELL ON THE PRODUCT DETAIL PAGE, THEN SALES WILL INCREASE BECAUSE THIS PLACEMENT MIMICS THE IN-STORE BUYING EXPERIENCE.
  37. 37. VARIATION MVMT WATCHES @CaraHarshman #MozCon@CaraHarshman #MozCon
  38. 38. RESULTS @CaraHarshman #MozCon CONVERSION LIFT DESKTOP: 2.2% MOBILE: 5.5%
  39. 39. @CaraHarshman #MozCon
  40. 40. A/B TEST HACKATHON @CaraHarshman #MozCon 1. GIVE PEOPLE ACCESS TO DATA 2. INVITE CROSS FUNCTIONAL TEAMS 3. GIVE PRIZES 4. FOLLOW UP
  41. 41. TESTING HACKATHON POSTER DOWNLOAD OPTIMIZELY.COM/TESTING-TOOLKIT/
  42. 42. @CaraHarshman #MozCon X X ANALYTICS VOICE OF THE CUSTOMER IDEAS HYPOTHESES
  43. 43. EXECUTION AUTOMATION EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon GET SHIT DONE
  44. 44. EXECUTION AUTOMATION @CaraHarshman #MozCon GET SHIT DONEGET SHIT DONE
  45. 45. EXECUTION AUTOMATION @CaraHarshman #MozCon GET SHIT DONE MAKE A PLAN SO WE CAN SCALE
  46. 46. @CaraHarshman #MozCon GET SHIT DONE X X PRIORITIZATION STANDARDIZATION EXECUTION AUTOMATION
  47. 47. IF YOU WANT TO SCALE, THEN YOU HAVE TO TAKE OFF YOUR GET-SHIT-DONE-HAT Image source: http://teeeshop.com/products/get-shit-done-bucket-hat
  48. 48. PRIORITIZATION @CaraHarshman #MozCon AN EMOTION-FREE SYSTEM TO DECIDE WHAT TO TEST
  49. 49. PRIORITIZATION CHART: EASE VS. IMPACT @CaraHarshman #MozCon IF YOU CAN MAKE TIME DEPRIORITIZE NEEDS A VERY SOLID HYPOTHESIS DO IT! EASE IMPACT
  50. 50. PRIORITIZATION SCORE CARD @CaraHarshman #MozCon RULE 1 point 0 points page part of funnel outside funnel audience 100% of visitors specific audience success metric primary goal secondary goal A RULE & POINT BASED SYSTEM WHERE EVERY HYPOTHESIS GETS A SCORE
  51. 51. “IF YOU COME TO ME WITH AN IDEA AND IT’S NOT LIVE IN 2 WEEKS, IT’S NOT BECAUSE IT’S A BAD IDEA – IT’S BECAUSE I HAVE BETTER THINGS TO TEST.” -Pauline, PM at Hotwire
  52. 52. PRIORITIZATION WORKSHEETS DOWNLOAD OPTIMIZELY.COM/TESTING-TOOLKIT/
  53. 53. STANDARDIZATION @CaraHarshman #MozCon A BROADLY AGREED-UPON SET OF PRIORITIES, METRICS, DEFINITIONS
  54. 54. 3 PIECES TO STANDARDIZE @CaraHarshman #MozCon 1. MAIN SUCCESS METRICS 2. STOPPING CRITERIA FOR TESTS 3. IMPORTANT AUDIENCES
  55. 55. STANDARDIZE SUCCESS METRICS @CaraHarshman #MozCon WHAT ARE THE ACTIONS YOU ARE TRYING TO MOVE? COMPANY GOALS TEAM GOALS TESTING GOALS TEST HYPOTHESIS GOAL
  56. 56. STANDARDIZE STOPPING CRITERIA @CaraHarshman #MozCon HOW LONG WILL A TEST RUN BEFORE YOU STOP IT? STOP
  57. 57. STANDARDIZE YOUR AUDIENCES @CaraHarshman #MozCon WHICH UNIQUE TRAFFIC SEGMENTS DESERVE MORE ATTENTION? New Google Mobile Desktop Returning Referral
  58. 58. BROOKS RUNNING A STANDARDIZATION CASE STUDY
  59. 59. HYPOTHESIS BROOKS RUNNING @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES,
  60. 60. HYPOTHESIS BROOKS RUNNING @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES, THEN THEY WILL BE LESS LIKELY TO RETURN
  61. 61. HYPOTHESIS BROOKS RUNNING @CaraHarshman #MozCon@CaraHarshman #MozCon IF WE DELIVER UNIQUE CONTENT TO SHOPPERS BUYING SIMILAR SIZE SHOES, THEN THEY WILL BE LESS LIKELY TO RETURN BECAUSE THEY DECIDED ON A SIZE.
  62. 62. @CaraHarshman #MozCon EXPERIENCE FOR INDECISIVE SHOPPERS
  63. 63. RESULTS @CaraHarshman #MozCon RETURN RATE FOR THIS AUDIENCE: -80% CONVERSION RATE: +2%
  64. 64. @CaraHarshman #MozCon GET SHIT DONE X X PRIORITIZATION STANDARDIZATION EXECUTION AUTOMATION
  65. 65. TEAM ADVOCATES EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon
  66. 66. TEAM ADVOCATES @CaraHarshman #MozCon PEOPLE WHO DO WORK
  67. 67. TEAM ADVOCATES @CaraHarshman #MozCon PEOPLE WHO DO WORK PEOPLE WHO PROMOTE YOUR WORK
  68. 68. @CaraHarshman #MozCon X X COMMUNICATION DIVERSE SKILLS TEAM ADVOCATES
  69. 69. 30% 29% 20% 21% WHO DOES OPTIMIZATION AT YOUR ORGANIZATION? Source: ConversionXL (PRETTY) GOOD NEWS @CaraHarshman #MozCon NOBODY A TEAM OF PEOPLE ONE PERSON PEOPLE ACROSS TEAMS
  70. 70. TOP 5 SKILL SETS TO HAVE @CaraHarshman #MozCon 1. PROJECT MANAGER 2. DEVELOPER 3. DESIGNER 4. ANALYST 5. FEARLESSNESS
  71. 71. 2 TEAM EXAMPLES @CaraHarshman #MozCon JOSEPHINE AT HOTWIRE DESIGN DEVELOPER CUSTOMER SUCCESS PRODUCT OWNER, HOTELS PRODUCT OWNER, FLIGHTS CRAIG HEAD OF GROWTH JOSEPHINE PRODUCT MANAGER FOR TESTING CRAIG AT EARNEST
  72. 72. COMMUNICATION CHANNELS @CaraHarshman #MozCon 1. OPT-IN EMAIL NEWSLETTER 2. WEEKLY MEETINGS FOR CORE CREW 3. INTERNAL WIKI / WORKFLOW TOOL 4. LUNCH N’ LEARNS 5. ALL HANDS MEETINGS
  73. 73. RESULTS SHARING TEMPLATES + DECKS DOWNLOAD OPTIMIZELY.COM/TESTING-TOOLKIT/
  74. 74. IDEAS HYPOTHESES EXECUTION AUTOMATION TEAM ADVOCATES EVOLVE YOUR A/B TESTING @CaraHarshman #MozCon
  75. 75. 1. LEARN RATE = 100% 2. BUSINESS GROWTH, HIGHER REVENUE 3. DELIGHTED CUSTOMERS 4. STRONGER DECISIONS & CULTURE OPTIMAL OUTCOME OF A/B TESTING EVOLUTION @CaraHarshman #MozCon
  76. 76. C A R A H A R S H M A N c a r a h a r s h m a n . m e @ c a r a h a r s h m a n

×