Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Culture of Optimization


Published on

No e-commerce site is perfect, and there will always be pressure to improve conversions and AOV. A/B and multivariate testing can be an incredibly effective tool when used strategically. Where do you start? What tools do you need? How can you get buy-in from all brand stakeholders? We'll give you the rundown.

Published in: Business
  • Login to see the comments

  • Be the first to like this

Culture of Optimization

  1. 1. PRESENTED BY: ERIC HANSER – DIGITAL OPERATIVE Culture of Optimization Ecommerce A/B and Multivariate Testing Primer
  2. 2. Tweet us @digitaloperativ
  3. 3. Tweet us @digitaloperativ
  4. 4. THESE ARTICLES WILL NOT GIVE YOU A STRATEGY Some of these tests may be effective, and some may not. Probably more of the latter. Every site is unique and has unique users with unique problems. Without more concrete hypotheses, a failed test leads to a dead end. Tweet us @digitaloperativ
  5. 5. NUMBER 1 Getting your site into shape is just like getting yourself into shape. There is no magic pill or simple solution. It is a lifestyle change. Tweet us @digitaloperativ
  6. 6. OK. SO WHERE DO YOU START? Tweet us @digitaloperativ
  7. 7. LET’S TAKE ONE STEP BACK. WHAT DO WE MEAN BY “TESTING” Tweet us @digitaloperativ
  8. 8. SPLIT TESTING: SIMPLE IDEA, LOTS OF POSSIBILITIES A split test is a controlled, randomized experiment run on your live site. Users to your site are split up and shown different versions of the site. The users’ progress is tracked to determine which version will be most successful. Tweet us @digitaloperativ
  9. 9. SPLIT TESTING: PLATFORMS Tweet us @digitaloperativ
  10. 10. CRO = CONVERSION RATE OPTIMIZATION Tweet us @digitaloperativ
  12. 12. THE TERM CRO CAN BE SOMEWHAT MISLEADING A “conversion” technically refers to a a user completed any desired action. In ecommerce, the word “conversion” is almost always used in reference to making a purchase. Tweet us @digitaloperativ
  13. 13. THE TERM CRO CAN BE SOMEWHAT MISLEADING Split testing can be used effectively to help with everything from branding to design to marketing, etc. By using the term “CRO,” some departments or colleagues might not see immediately that these tools and processes are relevant and helpful to the work they do. Tweet us @digitaloperativ
  14. 14. NUMBER 2 In building a culture of optimization, it may be helpful to limit how often you use the term “CRO.” Tweet us @digitaloperativ
  16. 16. WHAT ARE THE KINDS OF TESTS WE’RE TALKING ABOUT? Tweet us @digitaloperativ
  17. 17. A/B TESTS: SIMPLEST (AND RAREST) A/B Testing simply means testing one element on a page. Each option is called a version or a variation. Not limited to 2 variations. Sometimes called A/B/C or A/B/N testing. Only a single element is tested: color, placement, wording, picture, etc. If more than one of these elements change, it is no longer an A/B test. Tweet us @digitaloperativ
  18. 18. TYPES OF TESTS: A/B TESTING Tweet us @digitaloperativ
  19. 19. MULTIVARIATE TESTING: A BIT MORE TIME When more than one element on the page is being tested. Each possible combination of elements should be tested. With more variations, it can take longer to get statistically significant results (more on this later). All variations are still on a single page. Tweet us @digitaloperativ
  20. 20. TYPES OF TESTS: MULTIVARIATE TESTING Tweet us @digitaloperativ
  21. 21. MULTI-PAGE TESTING: USERS SEE MORE THAN ONE PAGE If an element appears on more than one page, you must test the change on each page. Also often called “Flow Testing.” Multi-page testing may be either A/B or Multivariate. Tweet us @digitaloperativ
  22. 22. TYPES OF TESTS: MULTI-PAGE TESTING Tweet us @digitaloperativ
  23. 23. SPLIT TESTING TYPES A/B Tests – A single-page test with a single element changed Multivariate Tests – A single-page test with more than one element changed. Multi-page Tests – A test that shows a change across more than one page in the user flow. Tweet us @digitaloperativ
  24. 24. ALL OF THESE TEST RANDOMLY SPLIT USERS INTO GROUPS. Tweet us @digitaloperativ
  25. 25. TARGETING: ALL USERS ARE NOT THE SAME. Optimizing your site for all users is important, but speaking directly to specific segments is even more effective. Many of the same tools that are used to run split testing can also be used to target specific audiences and optimize the site for them. Tweet us @digitaloperativ
  26. 26. NOT ALWAYS A TEST: SEGMENTING AND TARGETING Tweet us @digitaloperativ
  27. 27. NUMBER 3 Results from testing can lead to targeting. When looking at results of your test, you may find that certain segments should be targeted differently. Tweet us @digitaloperativ
  28. 28. SO, WHERE DO YOU BEGIN? Tweet us @digitaloperativ
  29. 29. IT IS TEMPTING TO JUMP RIGHT IN It may seem that there are things “wrong” with your site or that you can make tests based on “best practice.” Tests have “winners” and “losers” - there will be scrutiny. The more information you have, the better chance you have of making wins early on. Tweet us @digitaloperativ
  30. 30. NUMBER 4 Don’t start with testing! If your first few tests are flops, confidence in the whole idea of split testing could be compromised. Tweet us @digitaloperativ
  31. 31. START WITH LEARNING THE “WHATS” AND “WHYS” OF YOUR SITE. Tweet us @digitaloperativ
  32. 32. FIND THE “WHAT” DATA, DATA, DATA! Tweet us @digitaloperativ
  33. 33. FIND THE “WHAT” What are users doing or not doing? What do they like and/or dislike? Learn all you can from analytics. Proper set up and tracking could take time. Ask questions like: What are the biggest drop-off and exit points? What pages are most successful? What are the most optimal site flows? What are the least optimal? Tweet us @digitaloperativ
  34. 34. FIND THE “WHY” USABILITY TESTING Tweet us @digitaloperativ
  35. 35. FIND THE “WHY” With data collected from analytics, develop paths or a series of tasks for users to complete in a usability test. Several tools are available to either conduct in-person or remote usability testing. is one of the most popular. Often, unexpected and surprising results surface that you would have not necessarily seen in the analytics alone. Tweet us @digitaloperativ
  36. 36. YOU KNOW THE PROBLEMS. NOW, MAKE A HYPOTHESIS. Tweet us @digitaloperativ
  37. 37. START WITH THE HYPOTHESIS – THEN DEVELOP THE TEST When making a test, base your hypothesis on the problem. “If I change X, then Y should improve.” Never make a test and THEN ask the question, “how will I know which variation won?” Simply looking at overall ecommerce conversion will likely be inconclusive. Tweet us @digitaloperativ
  38. 38. WHAT & WHY 30% of traffic direct to product pages 63% bounce rate “Best Practice” recommendation to increase the add-to-cart button size and the free shipping message Tweet us @digitaloperativ
  39. 39. WHAT & WHY Users had no validation that the product was right for their specific condition. Users had no clear way to find the right brace for them. Tweet us @digitaloperativ
  40. 40. GETTING RESULTS STATISTICS, BLEH! Tweet us @digitaloperativ
  41. 41. GETTING RESULTS – DETERMINING THE WINNER! Two primary metrics: Percentage Increase = How much a variation improved over original Confidence Interval = How confident we are that the improvement is not due to chance A 10% improvement with an 85% confidence interval basically means that there is an 85% chance that the test will result in a 10% improvement. 95% confidence interval = Statistically Significant Tweet us @digitaloperativ
  42. 42. GETTING THE RESULTS STATISTICAL SIGNIFICANCE 10% Improvement 85% Confidence Interval 10% Improvement 95% Confidence Interval 15% Improvement 75% Confidence Interval Original Tweet us @digitaloperativ
  43. 43. WIN, LOSE, OR DRAW Optimization is ongoing and iterative. Ultimately, if a test is based on data and usability, we’re always learning and improving. Tweet us @digitaloperativ
  44. 44. NUMBER 5 If your testing is always based on data, every test will be a win. If your tests aren’t based on data, even wins may be losses. Tweet us @digitaloperativ
  45. 45. WINS & LOSSES – WHEN A WIN IS NOT A WIN. If a test is implemented without a clear hypothesis and clear goals, a win can lead to a dead end. If you don’t have a clear idea as to WHY a variation won, we’ve got nowhere to go. Tweet us @digitaloperativ
  46. 46. WHEN A WIN IS NOT A WIN Tweet us @digitaloperativ
  47. 47. WINS AND LOSSES – WHEN A LOSS IS NOT A LOSS Similarly, we can learn a lot when a variation underperforms if the hypothesis was based on data. With new information in mind, we can look again at the problem with additional data to find other solutions. Tweet us @digitaloperativ
  48. 48.
  51. 51. WINS & LOSSES – WHEN A DRAW IS A WIN Occasionally, a test goal will be something less “tangible” than an increase in conversions. Some tests’ goals, for example, will be to bring consistency to a cluttered site, make copy more on brand, or to adapt to changing business requirements. Tweet us @digitaloperativ
  52. 52. WINS AND LOSSES WHEN A DRAW IS A WIN Tweet us @digitaloperativ
  53. 53. MAKE A PLAN – PRIORITIZE! Not everything can be tested at once. The P.I.E prioritization process helps put all possible tests in a digestible order. P = Potential / How big is the problem? I = Importance / How valuable is the traffic? E = Ease / How difficult will the test be to implement? Tweet us @digitaloperativ
  54. 54. MAKE A PLAN PRIORITIZE! WEBSITE TESTING Prioritization Test Name HYPOTHESIS Potential Importance Ease PIE AVG PIE Notes Product Page - Short Product Description By giving users feedback related to their path (filters, categories, search terms) they will be more confident in purchasing 8 9 7 8 High Traffic, High Bounce Rate, text-only implementation, but need use of audiences tool Cart / Global Nav - View Cart Design and placement of "view cart" CTA will bring more people back into the purchasing funnel if they leave the cart 6 8 9 7.666666667 Valuable traffic, but most users are directed to the shopping cart. Easy implementation Category - Subcategory/Filter Design By providing clarity about the differentiation between subcateory links and filters, potential UX issues will be mitigated 8 8 7 7.666666667 Noted problem/issue in usability. Use of filters shows intent, and therefor important. Checkout - Security Messaging Overstating Security may actually be harming conversion. Possibly finding a more optimal amount of messaging could improve it 5 9 9 7.666666667 No known issue, but hightly valuable traffic, and extremely easy test to implement. Product Page - Find Other Products Allowing users on a PDP to easily click off to other similar products if it not appropriate for their needs, we decrease bounce rates 8 7 7 7.333333333 Lots of traffic, high bounce rate, but the feature would be new, and therefore importancei is measured Global Nav - Elements and Design Users perform better when accessing injury categories as opposed to body type cateogirs 8 8 6 7.333333333 As a global element, importance and potential is high. Implementation will be slightly more than others. Tweet us @digitaloperativ
  55. 55. NUMBER 6 Prioritization not only keeps you organized, it helps your organization understand the breadth and importance of optimization and testing overall. Tweet us @digitaloperativ
  56. 56. CULTURE OF OPTIMIZATION WEBSITE FITNESS TIPS There is no magic pill – It’s a lifestyle change. The term “CRO” can be confusing – Use “Optimization & Testing.” Test then target – Some segments may need to be targeted differently. Don’t start with testing – Gather data (“whats” and “whys”) first. Base tests on solid data and hypotheses – All tests can be wins. Tweet us @digitaloperativ
  58. 58. PLEASE USE THE OR APP TO RATE THIS SESSION Thank you for attending!