Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Run Smarter A/B Tests, Faster

270 views

Published on

We’ll cover the ins and outs of running a high velocity A/B testing and experimentation program so that you can run the most meaningful tests as quickly as possible without sacrificing the quality of insights.

Published in: Marketing
  • Be the first to comment

How to Run Smarter A/B Tests, Faster

  1. 1. How to Run Smarter A/B Tests, Faster How to Run Smarter A/B Tests, Faster @shanelle_mullin We’ll cover… 1. How to conduct conversion research to find smarter A/B test ideas. 2. How to prioritize your A/B test ideas in a meaningful way. 3. What a high velocity testing program is and why it works. 4. Common high velocity testing program mistakes to avoid. 5. How to increase your testing velocity responsibly.
  2. 2. How to Run Smarter A/B Tests, Faster “You don’t have generic problems, you have specific problems.” ~Peep Laja
  3. 3. How to Run Smarter A/B Tests, Faster
  4. 4. How to Run Smarter A/B Tests, Faster 1. Relevancy Does the page’s content meet user expectation? Does the page’s design meet user expectation? How can it match what they expect / want even more?
  5. 5. How to Run Smarter A/B Tests, Faster 2. Clarity Is the content on this page as clear as possible? Is the value prop on this page as clear as possible? How can we make it clearer?
  6. 6. How to Run Smarter A/B Tests, Faster 3. Motivation What’s motivating the visitor to take action? Is it clear, believable, timely and persuasive? How can we increase motivation even more?
  7. 7. How to Run Smarter A/B Tests, Faster 4. Friction What is causing doubts, hesitations and uncertainties? What makes the process difficult? How can we simplify the process?
  8. 8. How to Run Smarter A/B Tests, Faster 5. Distraction What’s not helping the user take action? What’s drawing attention? Is everything that draws attention really necessary?
  9. 9. How to Run Smarter A/B Tests, Faster Cross-Browser Testing Audience > Technology > Browser & OS Switch from the default “Data” view to the “Comparison” view.
  10. 10. How to Run Smarter A/B Tests, Faster Cross-Browser Testing Always compare within the same browser family.
  11. 11. How to Run Smarter A/B Tests, Faster Cross-Device Testing Audience > Mobile > Devices Switch from the default “Data” view to the “Comparison” view.
  12. 12. How to Run Smarter A/B Tests, Faster Speed Analysis Page Load Time: Seconds until every element on the page is done loading. Page Interactive Time: Seconds until the site is usable.
  13. 13. How to Run Smarter A/B Tests, Faster Speed Analysis Your site loads in 3 seconds or less. Your site loads in 4 to 7 seconds. Your site loads in 7 to 10 seconds (or more).
  14. 14. How to Run Smarter A/B Tests, Faster Speed Analysis Behavior > Site Speed > Page Timings Switch from the default “Data” view to the “Comparison” view.
  15. 15. How to Run Smarter A/B Tests, Faster Google Analytics Health Check 1. Are you collecting all of the data you need? 2. Can you trust the data you are collecting? 3. Is anything broken or tracking / reporting incorrectly? Why?
  16. 16. How to Run Smarter A/B Tests, Faster Common Google Analytics Mistakes 1. Pages are missing. 2. Data is different in the shopping cart tool. 3. Cross-domain tracking isn’t working.
  17. 17. How to Run Smarter A/B Tests, Faster Full Health Check Guide bit.ly/gahealthcheck
  18. 18. How to Run Smarter A/B Tests, Faster Start with the Highest Value 1. High volume, low value pages. 2. Low volume, high value pages.
  19. 19. How to Run Smarter A/B Tests, Faster Finding Broken Links
  20. 20. How to Run Smarter A/B Tests, Faster Finding Broken Links
  21. 21. How to Run Smarter A/B Tests, Faster Auditing Your Funnel The further down the funnel, the smaller the necessary impact.
  22. 22. How to Run Smarter A/B Tests, Faster Auditing Your Funnel
  23. 23. How to Run Smarter A/B Tests, Faster Auditing Your Funnel
  24. 24. How to Run Smarter A/B Tests, Faster Exploring Internal Search
  25. 25. How to Run Smarter A/B Tests, Faster Segmentation Is the Secret bit.ly/gasegmentation
  26. 26. How to Run Smarter A/B Tests, Faster Click Maps 1. This info is largely available in Google Analytics via “Navigation Summary”. 2. See when people are clicking on non-links.
  27. 27. How to Run Smarter A/B Tests, Faster Scroll Maps 1. Better prioritize your content and messaging. 2. Drastic colour changes mean a need for visual cues… likely indicate false bottom.
  28. 28. How to Run Smarter A/B Tests, Faster Session Replays 1. Real users with real money. 2. Where do they pause? Where do they give up? 3. Use this to gather more insight on poor performers.
  29. 29. How to Run Smarter A/B Tests, Faster Form Analytics 1. Which form fields cause the most error messages? 2. Which do people hesitate to fill? 3. Which do people leave empty, even if required?
  30. 30. How to Run Smarter A/B Tests, Faster On-Site Surveys Exit surveys. On-page surveys.
  31. 31. How to Run Smarter A/B Tests, Faster On-Site Survey Questions 1. What was the purpose of your visit to our site? 2. Were you able to complete your task? 3. If not, why? 4. Is there anything holding you back? 5. Do you have unanswered questions? Focus on actionable data, on the friction.
  32. 32. How to Run Smarter A/B Tests, Faster On-Site Survey Options 1 open-ended question. Y/N and then an explanation post-answer.
  33. 33. How to Run Smarter A/B Tests, Faster On-Site Survey Guide bit.ly/onsitesurveys
  34. 34. How to Run Smarter A/B Tests, Faster Customer Surveys 1. First time buyers. 2. No previous relationship with you. 3. Try for ~200 responses.
  35. 35. How to Run Smarter A/B Tests, Faster Customer Survey Questions 1. Who are you? 2. What are you using this product for? 3. What problem does it solve for you? 4. What do you like about the product most? 5. Did you consider any alternatives? 6. Why did you choose us over the competition? 7. What’s the one thing that nearly stopped you from buying? 8. What was your biggest concern or fear about us?
  36. 36. How to Run Smarter A/B Tests, Faster User Testing Tasks 1. A specific task. 2. A broad task. 3. Funnel completion.
  37. 37. How to Run Smarter A/B Tests, Faster Constant Ideation 1. Continue conducting conversion research. 2. Involve the entire team. 3. Focus on the entire funnel (acquisition to referral).
  38. 38. How to Run Smarter A/B Tests, Faster “If you come to me with an idea and it’s not live in 2 weeks, it’s not because it’s a bad idea. It’s because I have better things to test.” ~Pauline Marol
  39. 39. How to Run Smarter A/B Tests, Faster The Prioritization Problem 1. Not objective enough. 2. Not data-driven enough. 3. Not easily customized.
  40. 40. How to Run Smarter A/B Tests, Faster
  41. 41. How to Run Smarter A/B Tests, Faster Benefits of PXL 1. Makes “potential” more objective. 2. Makes “ease” more objective. 3. Helps create a data-driven culture. 4. Easy to customize.
  42. 42. How to Run Smarter A/B Tests, Faster Prioritization Spreadsheet bit.ly/pxlmodel
  43. 43. How to Run Smarter A/B Tests, Faster Better Management 1. Review KPIs. 2. Review tests launched (and not launched). 3. Review insights gained. 4. Choose tests for the upcoming week. 5. Discuss resources (or lack thereof).
  44. 44. How to Run Smarter A/B Tests, Faster Archive (Or Live to Regret It) 1. Avoid repeating tests by mistake. 2. Easily communicate wins and insights. 3. Improve the quality of future tests. 4. Create an insights > wins culture.
  45. 45. How to Run Smarter A/B Tests, Faster High Velocity Mistakes 1. Without buy-in and a testing culture, you’re doomed. 2. Calling it too soon and non-representative samples. 3. Letting quality and effectiveness slip.
  46. 46. How to Run Smarter A/B Tests, Faster Testing Capacity (52 / Avg. Req. Test Duration) x Number of Testable Pages (52 / 2) x 10 = 260
  47. 47. How to Run Smarter A/B Tests, Faster Measuring Velocity 1. How well are you trending towards achieving your capacity? 2. Trends are what matters… are you getting closer? 3. Chart it.
  48. 48. How to Run Smarter A/B Tests, Faster How to Run Smarter A/B Tests, Faster @shanelle_mullin 1. Conduct proper conversion research. 2. Come up with ideas constantly. 3. Prioritize with data. 4. Manage the process. 5. Archive your learnings. 6. Achieve your testing capacity. 7. Stop listening to CRO BS you read online.
  49. 49. How to Run Smarter A/B Tests, Faster Resources Optimization Training Program bit.ly/cxlinstitute Original UX Research bit.ly/cxlresearch
  50. 50. How to Run Smarter A/B Tests, Faster Questions? Ask any time via email: shanelle@conversionxl.com

×