Top 10 Tips for Your Split Test Program by Justin Rondeau
•
0 likes•322 views
Justin runs all of the optimization efforts and split tests at Digital Marketer. A top-rated domestic and international speaker, Rondeau has spent his career working on optimization campaigns & has trained some of the leading optimization teams at Fortune 500 companies. Rondeau has run 100s of tests for both B2B and eCommerce brands and has analyzed 3,000+ tests across virtually every industry.
Recommended
Recommended
More Related Content
Similar to Top 10 Tips for Your Split Test Program by Justin Rondeau
Similar to Top 10 Tips for Your Split Test Program by Justin Rondeau (20)
More from Utah Digital Marketing Collective
More from Utah Digital Marketing Collective (20)
Recently uploaded
Recently uploaded (20)
Top 10 Tips for Your Split Test Program by Justin Rondeau
- 1. Top 10 Tips For Your Split Test Program
- 2. TIP 1: FOCUS ON PROCESS FIRST, NOT ‘HACKS’ @jtrondeau
- 5. Results: No Statistical Difference Confidence: N/A Takeaway: The extra field increased lead quality - so keep it!
- 7. Results: 31.3% Lift In Opt-Ins Confidence: 95%
- 10. @jtrondeau It all starts with identifying goals & KPIs!
- 11. @jtrondeau Top Funnel Mid Funnel Bottom Funnel Engagement metrics, not directly connected with business goals. Robust metrics, directly connected to business goals. Generally associated with lead generation or ‘relationship’ changing metrics.
- 12. @jtrondeau Page Goals Business Goals Immediate Conversion Clicks Opt-Ins RPV Sales AOV
- 13. @jtrondeau
- 14. @jtrondeau Protip: Just because a top funnel metric increased at x% doesn’t mean a deep funnel metric will increase at that rate (or increase at all).
- 15. @jtrondeau
- 16. @jtrondeau Results: 2.01% Drop in Product Views 4.43% Drop in Add-to-Carts 10.59% Drop in Orders
- 17. BENEFITS OF TESTING MULTIPLE METRICS @jtrondeau #1 GET THE FULL STORY MORE USEFUL METRICS, MORE INSIGHTS #2 CAN GET INSIGHTS ON LOW SAMPLES IF THERE IS CONSISTENCY BETWEEN PAGES/ METRICS YOU MIGHT BE ABLE TO CALL A TEST EARLIER #3 BETTER ITERATIONS TESTING IS AN ITERATIVE PROCESS, THE MORE DATA YOU HAVE THE MORE YOU CAN IMPROVE.
- 19. @jtrondeau
- 20. @jtrondeau
- 21. @jtrondeau
- 22. WHAT CAN I LEARN? @jtrondeau DESPITE THE SMALLER SAMPLE SIZE FOR THE DEEPER FUNNEL CONVERSION… …THE CONSISTENCY IN ACTIONS OVER 3 PAGES AND 8 DAYS MADE US CONFIDENT THAT THIS TEST WAS COMPLETE.
- 23. TIP 3: USE ANALYTICS TO IDENTIFY PAGES @jtrondeau
- 24. @jtrondeau
- 25. HOME CATEGORY PRODUCT CART THANK YOU EVALUATING YOUR FUNNEL WILL HIGHLIGHT YOUR PROBLEM AREAS DETAILED FUNNEL ANALYSIS IS YOUR TEST’S BACKBONE
- 26. UniqueViews(In1,000’s) 0 25 50 75 100 Page Viewed HOME CATEGORY PRODUCT CART THANK YOU
- 28. PROTIP @jtrondeau WE CAN SURMISE A LOT FROM QUANTITATIVE DATA… …WE DEVELOP HYPOTHESES WHEN WE EVALUATE QUALITATIVE DATA. (WE’LL TALK ABOUT THIS LATER)
- 29. @jtrondeau
- 30. PICK PAGES THAT MATTER @jtrondeau #1 HIGH VALUE PAGES WITH A HIGH BOUNCE RATE BOUNCES ARE BAD NEWS BEARS #2 DEEP FUNNEL PAGES PEOPLE JUMPED THROUGH YOUR HOOPS, BRING THEM HOME #3 FUNNEL VALLEYS CHECK THE PREVIOUS PAGE TO FIND OUT WHY NO ONE IS MOVING DOWN THE FUNNEL
- 31. OR MY PERSONAL FAVORITE @jtrondeau High Visit, High Bounced Pages
- 34. @jtrondeau
- 44. 57% Decrease In Time Spent On Field 96% Decrease In Refills (this is HUGE)
- 46. TIP 5: SCHEDULE A FINITE STOP TIME @jtrondeau
- 48. @jtrondeau
- 49. @jtrondeau Results: 6.49% Increase In Sales Confidence: 95%
- 50. @jtrondeau
- 51. @jtrondeau Results: Decreased Add To Cart Clicks by 6.6% For New Visitors. Confidence: 95%
- 54. @jtrondeau “Netflix considers 90% of what they try to be wrong” -- Mike Moran “80% of the time you/we are wrong about what a customer wants” -- Avinash Kaushik Microsoft’s statistic from thousands of controlled experiments: only 10-30% of experiments move the metrics they were designed to improve” — Ronny Kohavi
- 55. @jtrondeau FACTORS THAT IMPACT TEST TIMING #1 LIFT PERCENTAGE WHAT WAS THE LIFT OR LOSS #2 CONFIDENCE RATE 95% OR BETTER #3 NUMBER OF VARIANTS 2 OR MORE #4 NUMBER OF VARIANTS TESTED POPULATION PER DAY
- 56. @jtrondeau WORK BACKWARD #1 PICK TIME PERIOD 1-6 WEEKS #2 RECORD UNIQUE TRAFFIC WHAT’S THE TRAFFIC FOR THE TIME PERIOD #3 NUMBER OF VARIANTS DIVIDE TRAFFIC BY VARIANTS #4 DESIRED CONFIDENCE 95% #4 CALCULATE LIFT PERCENTAGE CAN YOU GET X LIFT? IF YES, DO IT!
- 57. @jtrondeau IF YES, THEN SCHEDULE (AND STICK TO IT!)
- 58. TIP 6: FOCUS ON THE USER @jtrondeau
- 59. CONVERSION RATE OPTIMIZATION ISN’T ABOUT INCREASING CONVERSION IT’S ABOUT CREATING AN EXPERIENCE THAT IS INTUITIVE TO YOUR USER.
- 60. WE NEED DETAILED USER RESEARCH @jtrondeau #1 HEATMAPS SEE WHERE PEOPLE ARE CLICKING/SCROLLING #2 USER SURVEYS GET INFORMATION FROM YOUR BEST USERS #3 SESSION RECORDINGS WATCH EXACTLY HOW PEOPLE INTERACT ON YOUR SITE #4 USABILITY TESTS GET SITE TESTERS AND ASK FOR FEEDBACK
- 65. @jtrondeau We Saw a 40.9% Increase In DM Lab Sales When We Added A Link To Our Sales Page
- 69. @jtrondeau SOME SWEET SURVEY TIPS #1 ASK A SINGLE QUESTION IF ASKING FOR MORE, THEN SET TIME CONSTRAINTS #2 OPEN ENDED = POWER YOU’LL GET LESS RESPONSES, BUT BETTER DATA #3 KNOW IT’S A SEGMENT THE TYPE OF PEOPLE WHO ANSWER SURVEYS AREN’T AN ACCURATE REPRESENTATION OF THE WHOLE
- 70. @jtrondeau • Was it was easy to find what you were looking for? • Were the words/vocab used to define categories/sub-categories clear to you? • Do you have any suggestions to improve our website navigation? • Does our website look credible to you? • Is our pricing clear? • Is there anything else you’d like to know before signing up with us? • Will you shop with us again? Why/why not? • Do you think the form has any confusing/unnecessary input fields?
- 71. @jtrondeau EXAMPLE OF A USER TEST
- 72. @jtrondeau MY FAVORITE TYPE OF QUAL. DATA COMMON CUSTOMER SERVICE QUESTIONS AND COMMON SALES QUESTIONS
- 73. TIP 7: TREAD LIGHTLY WITH CASE STUDIES @jtrondeau
- 74. @jtrondeau CASE STUDIES ARE FOR INSPIRATION ONLY They tested the hamburger icon - I should too even though my mobile traffic is at 1%!
- 75. @jtrondeau TIPS FOR CASE STUDIES #1 CONTEXTUALIZE THE STUDY IDENTIFY THE INDUSTRY #2 IDENTIFY TESTED SEGMENTS IS THIS THEIR ENTIRE MARKET OR A PART OF IT? #3 BE SKEPTICAL OF LIFT PERCENT IF YOU DON’T SEE RAW DATA, BE SUSPECT. IT’S EASY TO MANIPULATE DATA.
- 76. TIP 8: SEGMENT BUT DON’T OVER SEGMENT @jtrondeau
- 78. @jtrondeau
- 81. @jtrondeau BUT BE CAREFUL! OVER SEGMENTATION ISN’T SCALABLE WHAT WORKS FOR ONE SEGMENT, MAY NOT WORK FOR ANOTHER. THE MORE GRANULAR YOU GET… …THE SMALLER THE SAMPLE SIZE. THIS MAY DISQUALIFY YOUR TEST
- 82. “Respondents targeting more than 20% of their visitors with personalized content achieved a 5% conversion rate, which was 2x the average conversion rate of 2.5% in their study” [Adobe 2014 Survey]
- 84. WHETHER IT’S THE DAY OF THE WEEK OR A HOLIDAY SEASON, YOU MUST ACCOUNT FOR TIME! ONE OF THE MOST DIFFICULT VARIABLES IS TIME
- 85. @jtrondeau IT ALL COMES DOWN TO CONVERTING ACTIONS Minimum Number Of Conversions Per Day Test Period (Days) 2 Variants 3 Variants 4 Variants 5 Variants 7 29 43 58 72 14 15 22 29 36 21 10 15 20 24 28 8 11 15 18 35 6 9 12 15 42 5 8 10 12 49 5 7 9 11 56 4 6 8 9
- 87. @jtrondeau
- 90. @jtrondeau CONVERTED BETTER ON TUESDAY, WEDNESDAY, & THURSDAY AFTERNOON CONVERTED BETTER ON ALL WEEKDAY AND WEEKEND MORNINGS
- 91. ALWAYS COMPLETE THE WEEK… …ALWAYS THERE’S ONE RULE @jtrondeau
- 92. @jtrondeau WATCH OUT FOR INITIAL VOLATILITY
- 96. #1 I HAVE A RULE IN MIND. I WANT YOU TO GUESS THE RULE #2 I’LL SHARE A NUMBER SEQUENCE. YOU WILL EVALUATE THIS SEQUENCE #3 YOU PROPOSE A SEQUENCE. I WILL TELL YOU IF IT FOLLOWS THE RULE #4 PROPOSE AS MANY SEQUENCES AS YOU WANT. WHEN YOU THINK YOU KNOW MY RULE - TELL ME (YOU ONLY GET 1 SHOT) HERE ARE THE RULES @jtrondeau
- 97. - THE NUMBER SEQUENCE - 2, 4, 6@jtrondeau
- 98. - HERE’S THE RULE - ASCENDING NUMBERS @jtrondeau
- 99. “INSTEAD OF TRYING TO FALSIFY A HYPOTHESIS, PEOPLE TEND TO TRY TO CONFIRM THE HYPOTHESIS” THIS IS CALLED WASON'S RULE @jtrondeau
- 100. WE’RE GREAT AT RECOGNIZING PATTERNS! EVEN WHEN THEY AREN’T THERE!@jtrondeau
- 103. USE THE SCIENTIFIC METHOD 1DEVELOP HYPOTHESIS PROFIT 2TRY TO DISPROVE 3 SO WHAT NOW? @jtrondeau
- 104. FINDING WHAT DOESN’T FIT WE MUST GET BETTER AT LET’S TRY TWO EXERCISES @jtrondeau
- 105. @jtrondeau WHICH CARD(S) MUST YOU TURN OVER? RULE: IF A CARD HAS A CIRCLE ON ONE SIDE, THEN IT HAS THE COLOR YELLOW ON THE OTHER SIDE
- 106. THE RESULTS @jtrondeau 46% 33% 7% 4% 10% OTHER JOHNSON-LAIRD & WASON, 1970A
- 107. @jtrondeau WHICH CARD(S) MUST YOU TURN OVER? RULE: IF A PERSON DRINKS AN ALCOHOLIC DRINK, THEN THEY MUST BE OVER THE AGE OF 21 YEARS OLD. 22 YEAR OLD 17 YEAR OLD
- 108. WHICH CARD(S) MUST YOU TURN OVER? RULE: IF A PERSON DRINKS AN ALCOHOLIC DRINK, THEN THEY MUST BE OVER THE AGE OF 21 YEARS OLD. 17 YEAR OLD
- 111. TIPS FOR RUNNING TESTS 1PROCESS NOT HACKS ID WITH ANALYTICS MEASURE WHAT MATTERS FIX WHAT’S BROKEN 3 2 4 @jtrondeau
- 112. TIPS FOR RUNNING TESTS 5SCHEDULE TEST TIMELINE CASE STUDY SKEPTICISM FOCUS ON THE USER SEGMENT APPROPRIATELY 7 6 8
- 113. TIPS FOR RUNNING TESTS 9TIME AWARENESS AVOID BIASES10 @jtrondeau
- 114. THANK YOU!